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ABSIBACT 

This evaluation report investigates the effectiveness 
of the Unified Science and Hathematics for Elementary Schools (USHES) 
program during the academic year, 1973-74. USHES is a carricular 
program designed to develop the problem solving abilities of students 
attending grades one through eight. The goal of the program is the 
development of 32 interdiscipj.inary units engaging the student in 
long-range investigations of real and practical problems taken frpm 
his or her school or community environment, ihese areas were 
identified as high priority issues during the first year of a 
comprehensive program evaluation: actual usage,, proof of concept, 
materials, teacher training, and indirect effects. Once these issues 
were specified, the evaluators selected the following indicators 
through which they would gather the required data: (1) a test 
especially designed for assessing problem solving abilities; (2) 
selected subtests of the Stanford Achievement Test; (3) observatipnal 
scales for classroom activity analyses.; (4). questionnaires surveying 

how the teachers were- implementing the. USHES program; (5) direct 

interviews with USHES teachers, principals, district. level 
administrators, and leaders of district resource teams who would 
train USHES teachers; and (6) on site visits for unstructured 
observations and interviews to determine the actual patterns of 
program adoption or adaptation. Following an initial overview, 
chapters two through six address each issue, one by one. A concluding 
chapter then summarizes, offers specific conclusions, and makes 
recommendations fo? the future development and implementation of the 
OSHES program. (EC) 
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CHAPTER I 

AN OVERVIEW OF THE USMES PROGRAM AND 
THE USMES EVALUATION PROJECT 

This evaluation project has been assigned the task of investigating 
the effectiveness of the USMES program during the academic year, I973-74, 
USMES, the Unified Science and Mathematics for Elementary Schools , is a 
curricular program designed to develop the problem solving abilities of 
students attending grades one through eight* 

USMES Philosophy and Goals 

The following statement of the purposes- and intentions of the USMES 
program is drawn from two descriptive documents prepared by their Central 
Staff: The USMES Ckiide (May, 1974) and the USMES Systems Approach to 
Development, Widespread Implementation and Maintenance of a Real Problem 
Solving Program in Elementary Schools (March , 1974) • 

The goal of the USMES program is the development of thirty-two inter- 
disciplinary units engaging the student in long-range investigations of 
real and practical problems taken from his or her school or community en- 
vironment* By responding to these problems, called »»chal 1 enges , the stu- 
dent develops his problem-solving abilities, and does so in a manner that 
gives him an experiential understanding (learning-by-doing) of the problem- 
solving process, as well as the acquisition of its basic skills and concepts 

USMES intends to teach the cognitive skills and strategies of problemr 
solving as a new area of learning, and not merely as a new method or a 
new content within an already defined area. Furthermore, this program 
sees itself as interdisciplinary in nature, in that its presence- in the 
curriculum would support and facilitate the existing disciplines-- 
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mathematics, reading, etc. 

[USMES] will not fulfill every ^orrU-lvn and 
affective need; .... other, mo Pro- 
grams may be needed' to tet ^nal 
aspects of the discipline art' -n 

the cognitive range of cliii i; ..trades 1-8. 

(The USMES Guide, p. 9.> 

USMES developers further believe that, to learn the process of 
problem solving, the student himself must analyze the prob^m: choose 
the variables to be investigated, search out the facts, idge the 

correctness of the hypotheses and conclusions. The teacher acts only 
as a coordinator and collaborator. This, they acknowledge, requires a 
new, more indirect style of teaching. 

progress toward a solution to a problem requires the combined efforts 
of a group of students, not just an individual student working alone. 
While some work may be done individually, the USMES construct provides 
for a division of labor and an exchange of ideas--a total group effort. 

A final essential characteristic of this program is the relevancy of 
the task. The "challenges" undertaken by the students must be both r;eal, , 
i.e., embody some valid aspect of school or community life rather than an 
invented problem imposed prepackaged by the curriculum, and practical, i.e. 
the student's solution may lead to the actual improvement of that situation 
being investigated. The problem leads to an experience of useful accomp- 
lishment' in the student's life. 

Evaluation Project Design 

When designing this project, the evaluators reviewed the informational 
needs of a variety of audiences: the National Science Foundation which 
sponsors both the program development and its independent evaluation; the 

13 
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developers of the USMES program and the members of their Planning Committee; 
the' on-site users of USME%''aVid ti^ain'e^^ 
and trainers. 

Our preliminary conversations with these interes oups led to the 
identification of several critical areas for invest. ^ 'tioh. These areas 
were identified as high priority issues during the first year of a compre- 
hensive evaluation of the program. We have defined these issues as follows: 
(a) Actual U sage. In those school situations where the USMES 
program is being implemented, what learning activities are 
actually occurring?" What student behaviors are being de- 
veloped as a result? What kind of student-to-student and 
student-to-teacher, interaction patterns are fostered under 
the USMES environment. How do these interaction patterns - 
differ from those of the "control" groups? 
(b) Proof of Concept . Have the problem solving abilities of 
' the students increased as a result of using USMES? Has 
this program affected in' a positive or negative manner the 
students' basic skill development, especially in reading 
and mathematics? These are seen as two interdependent 
issues. While NSF is concerned that the program's primary 
goal, the enhancement of problem solving, be actualized, 
professional educators (principals and teachers) are e- 
qually concerned that they'remain successfully accountable 
for the communication of basic skills. 
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(c) Ma terials . Are the supportive materials offered by USMES 
^ ^ ^beingMUsed^.',. Are. they, helpfy 

teachers' manuals, »'How-.To'» cards, design labs, and tech- 
nical papers. % 

(d) Teacher Trai^^^^ ^ the national USMES workshops effec- 
tively train - .uiiers? Moreover, does the Resource 
Personnel Workshop model prove to be an effective dissem- 
inator of the program? Are the present local training 
efforts capable of realizing a second generation of teachers 
and thereby insuring the continuation of this program? What 
further continuing support will be required? 

(e) Indirect Effects . Is the USMES program bearing signifi- 
cant secondary effects on the environment, e.g. on student 
attitudes. ..on teaching styles? Are there evidences of 
tertiary effects on those only indirectly related to the 
program: on colleagues of the involved teachers. .. on other 
students within the school. ..on the administrators of selected 
schools. ..on school scheduling. .. on school practices? 

Once these issues were specified the evaluators selected the following 
indicators through which they would gather their required data: 

(a) a test especially designed for assessing problem solving 
abilities. 

(b) selected subtests of the Stanford Achievement Test. 

(c) observational scales for classroom activity analyses. ^ 

(d) questionnaires surveying how the teachers were implementing 
the USMES program. ^ 

15 
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(e) direct interviews with USMES teachers, principals, 
district level administrators, and leaders of district 
resource teams who would train new USMES teachers. 

(f) on site visits for unstructured observations and inter- 
views to determine the actual patterns of program adop- 

! a 'Station. 

Evaluation Report 

The plan of this report is to draw on the various data products by 
these six indicators, to tabulate and interpret their results, and so to 
address the five central issues raised above. The report will proceed in 
this manner: following this initial overview, chapters two through six 
will address each issue, one by one. A concluding chapi^^v^ ^^^^ ^^^^ 
marize, offer specific conclusions, and make recommendations for the fu- 
ture developmen 3.nd implementation of the USMES prograffK. 

This reporr:.«.ll draw upon preliminary reports alre^ad* [»resented to 
the National Sciience Foundation and other audiences. How vex, their repre- 
sentation in this document will be adapted to the present concerns with new 
information added, and unnecessary duplication eliminated. 

In the coming weeks, a brief, summary document will be made available 
for distribution. This document will seek to address a wider audience, who 
are more inteiested in the basic inif-'ormation, summative conclxisions and 
recommendatio^s^;.•^ and less concerned; with the detailed tables commentary on 
the informati^>^ contained herein* 
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Caveat 



-For a correct appraisal of this report, one serious limitation must 
be- noted by the reader from the outset. This evaluation project was to 
have begun in August, 1973. However, it remained without funding until 
March 22, 1974. As a result, the evaluation team was unable to assume 
early control of the evaluation acCivities, to train observers, designate 
samples, advise observers on data collection problems, and monitor the 
data collection procGSS--all necessary to insure complete, usable rejiurns. 

To aid us in this limitation, the USMES Central Staff themselves under- 
took the responsibilities of sampling, observer draining, instrument pur- 
chase and pre-testiK:;^ Wbile their exHisnded efforts were admirable, it is 
obvious that such a :.^v -diuit e does not enhance the objectivity of an out- 
side evaluation and ta^t tnH does make presumptions on the time, experience 
^ and expertise, of the developers which can not be Jus^tified. 

These problems lesulttd in a serious loss of usable data. In some geo- 
graphic areas, observ^^^rs aid' not meet their commitments to collect data 
at the time of pre-t ^ting, or post-testing, or both. Some of the data 
were unanalyzable ber^vfe^i^ oi inappropriate testing procedures (e.g., chil- 
dren were proded in ^ , <»l) Um-solving tests; time Umi t.:v;vrere not followed^ 
the wrong SAT subtesnu wer.c administered). 

The data losses i ^-^t: damaging to an adequate assessment of student 

perfoimance in basic ski Lis and in problem solving because the measures for 
these traits were time co.v umg or otlierwise difficulc to administer, and 
they were disruptive t-. tth^> school day* 
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On the other hand, the schedule for site visitations and interviews 
was little effected by this eight month lapse for the grant award. It 
is the judgement of the evaluation team that other data, especially the 
interview data and the unstructured observations during the site visits, 
provided the most comprehensive, helpful, and illuminating information 
about the USMES program. We have relied heavily on these kinds of data in 
the assessments that follow. 
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CHAPTER II 
METHODS AND PROCEDURES 



ERIC 



The evaluation design for this project could be carried out in its 
original conception. Those compromises on the initial design which were 
necessitated by the late funding are described in this chapter. However 
the principle burden of this section is to describe to the reader the 
actual sample selections achieved, the data collection instruments appli< 
to these samples, the methods used for data collection, and those tech- 
niques of analysirs' which were employed. ^ j.^— 

Selection of Sample Classes 

USMES Teachers received various category designations according to 
the nature of their training and the extent of their USMES experience. 
The following types pertain to the 1973-74 evaluation program. 

(a) Development Teachers : Those who attended more than 
one national level workshop conducted by the Central 
Staff and who were expected to try out ideas for new 
USMES challenges in their classrooms. They were ex- 
perienced USMES teachers. 

(b) Implementation Teachers : Those who were new to USMES, 
who were attending their first national level workshop 
during the summer of 1973, either at Lansing, Michigan, 

or at Boston, Massachusetts. There they received training 
and resource materials for newly developed units. Their 
implementation of these new units were supposed to enable 
the developers to assess the adequacy of the workshop . 
training and utility of t±e resource materials. 
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(c) Locaii:fr Jj^jj- gj- -^^^ Te achers; "Second-generation" 
USMES teachers, trained -at the local level by District 
Resource Teams who used the Resource Personnel Workshop 
model. The team members had been trained at a national 
workshop conducted in Lansing, Michigan by thu USMES 
Central Staff with the help of consultant-specialists* 
In turn, the teams were expected to train and support 

a "second-genexa.tion*' of USMES teachers in their own 
districts. Tb£ intent was to implement a more cost 
effective metfesd of training teachers. 

(d) Chicago_ Works hx)p Train ed Teachexs; Those Area A Chicago 
teachers who attended the Chicago Workshop conducted 

by the Central S:taff in August, 1973. The purpose of 
this Chicago Dislirict Implementation Experiment was 
to investigate the strategies, support mechanisms and 
resources designexl to provide teacher training, program 
coordination and implementation in a city school dis- 
trict which was solely dependent on its own talents to 
achieve those ends. 
The proposed design for the 1973-74 evaluation called for a variety 
of data collections from samples of USMES teachers and their students in 
each af the four categories listed above. The number of USMES sample 
classes chosen from each category are shown in Table 1. 

Random samplirng: was not feasible; selections had to be limited to 
classes in those areas where trained observers were available for data 
collection. Furthermore, the cnvaliitation staff could not visit the many 
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TABLE I 



Sample Classes Selected for Data Collection and Sample Classes 
from Which Data was Obtained 



jjl _ Lll. II I >l I I'll ■' .. . 



Number of USHES Trained-Teachers 



National Development 
Sample 

National Implementation 
Sample 

Chicago District 
Implementation Sample 

Sample of Locally 
Trained RPW Teachers 



Selected 
for 

Sample 



Used 
USMES 



25 



18 



24 



28 



15 



12' 



i 



Confirmed! 
Did mL Use 
U3ES 
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No 

Information 
Obtained 



Number, Ox: Control Teachers 



Selected 
for 
Sample 



16 



18 



12 



Some 
Information 



No 

Information 



'These teachers clamed either to toe just begun a unit or to have planned introducing a unit 
chdlenge to their -classes within-a month of the interview-date... _ „: 
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widely scattered sites which could have resulted from a random sample. 
Given these constraints, purposive sampling was done to insure that the 
class selections represented a cross section of USMES unit challenges, 
■grade levels and socio-economic levels in a manageable number of geo- 
graphic areas. The number of sample classes in each of the geographic 
a^eas was proportionate to the intensity of USMES involvement expected 
by the program developers. 

Control class'e=a=wre-selected for only two of the four categories - 

of USMES teachers. Since the unit developnvent activitives and the de- 
velopment of local training strategies were formative in nature, no con- 
trols were selected for USMES development teachers or for RPW teachers. 
Control teachers were identified for each of the 18 national implementation 
sample teachers, as shown in Table 1. Controls were also selected for 
tUe Chicago implementation sample, but in this case, proportional n's 
were sought-12 controls for 24 Chicago USMES classes-to maximize salient 
information and minimize costs for data collection. 

With few exceptions,' the control classes were chosen from non-USMES 
classes'in the same schools as the sample USMES classes. If possible, 
controls were matched with USMES teachers on grade level and teaching 
experience. Members of the USMES Central Staff made the control selections 
and secured permission for the necessary evaluation activities. 

By the time the evaluation team was able to assume control of the 
evaluation project in March, 1974, serious data losses were beyond re- 
claim. The only appropriate or feasible time for collecting such infor- 
mal on as student performance data had gone by. Table 1 indicates the 

23 



-12- 



mimber of classes from which any data--student performance, teacher 
interviews, or program monitoring--was obtained. 

Characte r i st ies of Sa m pje USME S Sch ool s 

The descriptions of USMES sample schools presented below are based 

on data from the School Information Form shown in Appendix A. Because 

evaluators assured participating schools of anonymity, the exact locations 

of these schools are not indicated. 

A. Geograp hi cal Dist ri bution 

"USMES schools" are scattered throughout the country. That is, 

evidence of USMES usage, the presence of USMES materials, and/or the 

presence of USMES-trained teachers can be found in many sections of the 
country, in urban, suburban, and rural school systems, and in communities 
encompassing a variety of socio-economic levels. 

Program dissemination, however, has been most apparent in college and 
university towns where USMES Planning Committee Members or other USMES 
contacts reside and in towns located near the offices of the Program 
Developers. The former group of towns included Durham, New Hampshire; 
Athens, Georgia; East Lansing, Michigan; Iowa City, Iowa; Boulder, Colorado 
Bakersfield, California; and Monterey, California. The latter group of 
USMES locations--those areas near the development of f ices--included 
Arlington, Lexington, Watert^wn and Roxbury, Massachusetts. Even from 
the initial stages of program implementation, however, the developers 
also made continued efforts to disseminate the program in many less af- 
fluent urban settings: Atlanta, Georgia; Washington, D.C.; Lansing, 
Michigan; Chicago, Illinois; Minneapolis, Minnesota; and Los Angeles, 
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California; as well as Roxbury, Massachusetts. Only two states had more 
than a limited pocket of towns using USMES: Michigan and California. The 
above list is not intended to be exhaustive of the locations where USMES 
was used or was expected to be used during 1973-74; it is presented to illus- 
trate the wide pattern of geographical dissemination. 

Sample classes from which data was to be gathered for this Evaluation 
came from 37 schools. Chicago schools were to have been heavily represented 
in this sample of (13 of 37) because of the intensive district implementation 
experiment planned for Chicago. 

USMES observers in the sample site areas were directed to complete a 
.School Information Form for each of their sample schools. Completed forms 
were received from 29 of the 37 schools designated for the sample. Observers 
from the remaining eight sample schools did not meet their commitments for 
any data collection, and no information was obtained. While 13 of the 29. 
returns were from Chicago, only three of these Chicago schools had teachers 
who had used USMES at all during the year. The discussion below, therefore, 
is based on 19 schools: 3 from Chicago and 16 from scattered areas throughout 
the country. 

B. Population Densities of Communities 

The community settings of the 19 sample USMES schools can be character- 
ized as follows: 2 rural, 12 suburban, and 5 urban. This distribution 
seemed to reflect the national picture of USMES usage. Most of the school 
systems interested in trying USMES were suburban. Urban systems had too 
many other needs and immediate problems which preempted attention to USMES. 
Despite an intensive effort to disseminate USMES in Chicago Area A, only 
four teachers in three Chicago schools used the program at all during 1973-74. 
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C. Socio-economic Levels 

The coitununities in which the sample USMES schools were located rep- 
resented a wide cross section of socio-economic levels. These were dis- 

tributed as follows: 

Socio-econom icJ^evel WnmhPr of Sample Schools Reporting 

High 0 

Upper Middle 2 

Middle ^ 



Lower Middle 



5 



tI 

As indicated in Table 1, these completed forms were obtained from 15 
USMES development teachers and 13 implementation teachers-four from 
Chicago and nine from other parts of the country. We were able to secure 
12 completed forms from the Chicago Implementation sample and 9 from 
controls for the national implementation sample. No controls were desig- 
nated for the USMES development classes. 

This present report deals only with the data pertaining to teacher 
characteristics and class size. Further information on the nature and 

intensity of the treatments received by sample classes will be reviewed 

in Chapter III. 

1. Development, Teachers.. Class Information Forms were obtained for, 
15 USMES development teachers. The grade levels which they taught were 
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distributed as follows: 

Grade Levels . T eachers at that Grade 

■ . 3 4 
3-4 1 
4 3 



4-5 



1 



5 2 , 
4-6 1 

6 1 
6-7 1 

7 1 . 

The teachers of the class combining gi'ades 4 through 6 and of the class 
designated above as 6-7 were teachers in departmentalized schools. All 
other classes were self-contained. 

Glass size for these development teachers ranged from 17 through 30 
and averaged 27 students per class. 

The number of years of teaching experience represented by this sample 

of fifteen ranged from 3 to 20 years, with a median of 5.5. 

Most of these teachers had very little, if any, specialized training 
in math or science beyond pre-service methods courses and USMES workshops 
Their math/science training can be ouiranarized as follows; 

(a) 3 teachers had undergraduate math or science majors. 

(b) 1 teacher had taken one graduate course in math. 

(c) 2 teachers had taken one or more undergraduate courses 
in math and/or science. 

(d) 2 teachers had attended one or more in-service work- 
shops for other science curricula (SCIS, EES, SAPA) . 

(e) 1 teacher had attended an NSF summer institute. 
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(f) 6 teachers had no additional math/science training be- 
yond their undergraduate methods courses or USMES work- 
shops. 

2. Implementation Teachers . Of the first-trial implementation 
teachers who completed the Class Information Forms, four were from Chicago 
and nine were from other parts of the country. The grade level distri- 
bution for the 13 classes is as follows: 

Grade Levels Teachers at that Grade 

2 • 1 

2-3 1 . 

3 2 

5 2 

5-6 1 

6 1 

7 0 • 

8 2 

Both eighth grade classes were in departmentalized middle schools; the 
other classes were self-contained. 

Class size for the implementation teachers ranged from 16 to 31 with 
an average of 27 students per class. 

Further descriptive information on the implementation teacher sample- 
teaching experience, specialized math/science training, nature of non- 
USMES curricula used in their classes— is contained in the following 
comparison of these teachers with their control group counterparts. 
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3. Control Teachers , Class Information Forms were obtained from 
12 control teachers. Nine, of the controls were "matches" for the nine 
classes in the national USMES implementation sample. Three of the con- 
trols matched three of the four Chicago implementation classes. The 
missing' control class in this group was to have been a third grade, class. 
With this exception, the distribution of grade levels for control classes 
matched that shown earlier in this report for the implementation sample. 
Class sizes in the conti^ol sample were also "comparable in class sizes 
for the implementation sample. 

The USMES implementation teachers were similar to the group of 
control teachers with respect to both teaching experience and special- 
ized training in math or science. The USMES teachers had been teaching 
between 2 and 16 years, with 6.8 years as the median of teaching experi- 
ence. The range for the control teachers was between 3 and 17 years with 
a median of 7 years' experience. Two of the 14 implementation teachers 
and two of the 13 control teachers had undergraduate degree majors in 
math or science. None of the other teachers in either group had any 
special training in math or science beyond required math and science 
methods courses which were part of their teacher preparation programs. 

Selection of Interview Respondents 

Classes were to have served as the sampling unit, not only for test- 
ing student performance, but also for data collection via teacher inter- 
views and teacher questionnaires. Those teachers, whose classes were 
to have been tested, were also the teachers whom we intended to interview, 
and to whom we would mail questionnaires. However, since the need for 
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information on crittrcal issues about £utr^e USMES deveb^^nt and inmle- 
mcntation was -or-- nL. we propos.: - '-v ^v^d our efforts on I'h.ca coUsixtion 
via site visits. interviews, anc -.^^cher questionnaires-.techniques 

not jepordized ; ' 1 :e funding. 

Expanding r ><i-mple of interview respondents did not . .volve an in- 
crease in the rarr;-v-r of locations we would have visited 'u.j:in^ the orrxgiinal 
selection crit;erf.» . Sample teachers -±rom . the development --W-craiTied, 
and Chicago Works-op- trained teacher designations were already scheduled 
for interviews. However, when we ascertained that there were few locally- 
trained teachers doing USMES, and that very few Chicago teachers were using 
the program, we decided to interview implementation teachers and pre-service 
trained USMES teachers at those sites designated for visitations. 

The 80 USMES teachers, respondents to out interviews, can be classified 
according to the nature of their USMES training in the following manner: 

(a) 26 development teachers. 

(b) 16 implementation teachers. 

(c) 16 "second-generation" locally-trained, RPW teachers. 

(d) 20 Chicago Workshop trained teachers. 

(e) 2 pre-service trained teachers whose undergraduate 
teacher preparation in math-science educational methods 
included training in USMES. 

These 80 teachers encompassed a cross section of new and experienced USMES 
teachers at all grade levels. 



30 



.19- 



Addi'tional adnm-,i Aliiv- personnel were also interrriewed at these 
sites. Their roles .^x j.osiit.^ns, in respect to USMES, are identified 
as follows: 

(f) 13 R'r ~K,am headers responsible for organizing and 

con-i.' IVfn: \JSmS training workshops ax the local level. 
Of • Hi.. 4 were elementary school poincipals, 2 were 
ele.< i.,sr> ^:rhool teachers, 4 were district-level admi-n- 
ist: curriculum supervisors, and 3 were Elemen- 

tar\ Itir">rn Program supervisors from Michigan. Thus, 
all r-. four of the 17 District Resource Teams in attend- 
ance a:t tSie national Lansing Resource Personnel Workshop, 
were r-presented in the interviews. No site visitations 
were ms-d.s ot the Atlanta or La Grange, Georgia teams 
since uiiarr Leaders informed us by telephone that no 
local wr=r:3.hops had been conducted by May, 1974. Because 
of the rss?.' constraints on the evaluation team we were 
unable to schedule interviews with the FuUerton, Califor- 
nia team or with one of the Los, Angeles area groups, 
(g) 17 elementary school principals (in addition to the 4 
mentioTieii above serving as RPW team leaders). Of the' 
17, sevcBi haid attended a national USMES workshop. These 
pri' ipa]is rwere interviewed because they had, or were 
expuctii to, have. had, USMES teachers in their buildings. 
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(K) 3 district level curriculum supervisors, cc^iTiuiltants , 
* and resource teachers, all of whom were ser-i-n^ as 
members, but not leaders, of local RPW teams, 
(1) 5 district superintendents or associate sup^?. 
The following locations were represented by our intervi^eiW ^^pondeirt^sx 
Los Angeles, Bakersfleld, Oild^e, Monterey, Carmel, and CanpsfeiliL, Cal- 
ifornia; Boulder, Colorado; Minneapolis, Minnesota; Eaton Rap-cs, Lansing, 
Flint, Battlecreek, and Waren, Michigan; Chicago, Illinois; Vtesningtan., ED*C. 
Arlington, Lexington, Roxbury, and Watertown, Massachusetts; and Durham, 
and Gossville, New Hampshire. 

Se 1 ec tion of Questionnaire Respondents 

The Program Monitoring Form (Cf, Appendix C) was sent to TITSMES 
teachers in May, 1973. While all sample teachers received a copy of this 
form, it was determined at our on sxte visits that only 28 teachers from 
the original sample had sufficiently used USMES in their classes that yea^r 
to' be able to respond to the questionnaire in a meaningful manner. 

This questionnaire was also sent to an additional Jl USMES tesctiers 
from a list of narass and addresses supplied by the developers. At thexr 
i^equest, we agreed to expand the teacher sample using this form to secisre 
Hjuch needed information. This decision provided a much larger feta base 
with this relatively inexpensive method of data collection. 

Assessment of Student Performance in Problem Solving an d Basic Skills 

The USMES project seeks to enhance the problem solving abil±t±es <q€ 
elementary school students without impairing their basic skill dey/:eBkmmBmz:. 
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USMES claxESs. that^ by responding -trj creal life-, meaningful chall^l^^^"'^^ 
taken f rom ^zhe liical school,^ conunun-.ty envixonmetrt-, students will :dn- 
volved in ail st^ts of problem solving: aiiserwation, data collectm?!,' 
representac±an anil analysis of data, formni^atisirx and trial of siai^cBssive 
hypotheses, and decision on the f inal actran to ;be taken. The rpxoject 
further claims rthaJt while investig?aiting r:^- problems , students quickly 
learn many-ma thema tics-, science, social science., and. language arts skills. 

Valida.tion of this project concept necessitated the collectron of 
situdent performance data in the areas of both problem solving asai h^s±c 
skills. A pre-test, post-test control group design was pursued for Mj:th 
areas. Pjmblem soiviag skills were measured by the Playground :g!:^bfem, 
^ test sperrially designed for this USMES valuation. (Tiie Adminisrtraitio r ' s 
Manual and the Scoring Manual for the Playground Problem are f ouml^ in 
i^ippendicesIE and F.) Two subtests of the Stanford Achigvement Te ^^ 
Batter y were used to measure basic skills development iiL tinie critical 
areas of reading and mathema±:ics. Further ^ecification aad. disrcussion 
of the Instrumentation, data collection, scti:ring, analysis, and results 
of this data, on s.tiutdent perfermance will be found in Chapter IV. It ±s 
important: chat the reader review these resulits in the context of rour a^s^ 
cusslon on tiie ser:ious Linfestiorr of this data collection, which :is pro- 
vided in Chapter IV. 

Intervrewsiillhirigig -Site VlisitatioES. 

During-slte -alsits, the indivaficluals Liss^ earlier in takLs cSnapter 
were interviewed about those aspects of USMES appropriate ro the nr.e- 
spondent's; postion and experience with the program. The interviews took 
place over a four month period, extending fnrom February through May, 1974. 
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Incswiew gutifes -had been prepared ffof the largest respondent groups: 
USKES teachers, raam leaders, and pir-i-acipals. (Cf. Appendices G, H, 
and. 1, ressKccferaiy. ) 

We told a=^r^.mimdents at the ancsi; of the interviews that our purpose 
in speaking ^wdii^ them was, in essei=c£, to explore the feasibility of im- 
plementing nhs, OSMES program in va==ious schools, each with its unique 
set of assets ai*d limitations. We .ciisnphasized that our intention was to 
evaluate the program and its trainiaitg: component, not the respondent. We , 
trji-d to fusiter a /polite cordial, ncm-threatening interaction in order 
to elicit honest: respoiiises, both positive and negative. 

In geneial, car method of interviewing started with a broad question, 
followed by :f,uEth;er probing or requests for clarification on the issues 
which xesc^denirs raised. Xf there were any points not covered by a 
respon&ESs, for ^ch -^ wanted answers, we then asked direct questions. 

Whiiis; we entered ^e inte3==^w situations with preconceptions of what 
was impoxtrant to eva^kmte, Mt that it would be inappr.n^.iate to use 
a more .^irucrured tecbxiiqiiie, e^clally during this first year of a compre- 
-heiTfeisEr: iI^fiE. evaluation. SrrcctuEed Interviews force ;a choice between: 
rlgicffr^. fenasLated: arrsweis to rigSSUy formulated questions. 'While this 
techn-iapis -OTotes easy qnaaTifirfiGStfon, of responses, its use ipresumes that 
a pricnd- .^J^ificartiims resjmrses will reflect the important concerns 
of:.:the:EES35ondents. X^stead:, ;the ^ategy of intensive interviewing, with 
inte'rvlew =gniides enabled us to ask questions about what interested us, 
but also to: elicit from each respaimdent those concerns he considered to 
be of first Importance itt. the USMES program. ' Using this technique, we 



learn&d, for example, that problems with the Design Lab were of vital 
concern to teachers. The strength of diis feeling probably would not 
have been apparent had we used a prureLT objective IrsstrumenC* Moreover, 
our experience witJi cordial uv^tracitured incervaews has been rhat, :given 
a chance, meachers would talk f ir^sl: and at greatest length aibout those, 
aspects of t±e subject most meanxngmil to them. 

The usBi cEf unsitructirred intervriews was especially appropriate Cot 
gathering pertinent information oir lESMES teachsr txaining. The respondents 
were products of ;a variety of traainrcng models air Lad no USHES training at 
all. Of tho:se with USMES training, some were Erighl-y exi^rienced in its 
diassroom use, whiile others had not taught USMES at all... Local USMES 
training efforts were especially diverse, and we could not assume ^that ^ 
knew a great deal afcO'ut tlie reasons behind the prDdncttivrry or Inactivity 
of each team.. «e needed to :;i'-d out what kinds of things were£ happening 
at each site rather than to ; : tsncine the frequency of prf^determiit^d kinds 
of things thaa: we thougJ^t woidsa ia^e happtming. 

Most of. ciiE ^--n .p-r^-.v^.s; we re ^^nfiicted by members of 3±ae ^aibiEation 
team working iim pair^ On^rsember directed the questioning while the other 
served primari% as rec.£)rde:r. Excepi: as noted below, the i-re:spondents were 
interviewed individual ly. All interviewers were trained ijy the Project 
Director pr±or to.ssite visitations and were debriefed at group meetings of 
the evaluation team upon their return. 

Only 36 of aiJse 8i0 teasrhers wsere seen individually by these -two-peESom 
interview sssams-. :£2raup interviews ^were us.ed with the rssiniWg teachers: 
since they M J±ttL-e.£r.e.e time dnringi:the school day anst-'then were avaSil- 
able only in: groups- We were rel-ucrtant to impose on their time much .before 
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or after the hours of the normal school day. ISowever, only in one case 
did we feel that group interviewing was less productive than discussions-^^ 
with individual teachers. [in that case, ors- teacher dominarted the in- 
terview and stiff led the commenrs of her colleagues.] Generally, the 
small group situations evoked a more detralLei ::response from "the: teachers. 
These group interviews had the advantage of allowing the teachers more 
time to reflect, to recall ei^isnriences , to netihink and to amend initial 
accounts that upon reflectiam seemed in need of amplification., qualifica- 
tion, or revision. Overall, tfce tone remained cordiaU. ^ean ±n the group 
interviews, teachers felt fr^ disagree -t^itm, or qualify^ cojlie agues 
responses. 

Unstructured interviews pre^e^ntu two nsroiila:; ^r: (L) Tirxnxini^e 2iie sub- 
jectivity of the interviewer, ^*nd (2) px^eEr the Infermstrccrr, obrtained in 
some meaningful fxcrm. To stamdardxsie txie ±-:t£jrviewing pxoxi^res as much 
as possible, all ±nt ervrei^ers >xhes traizi^ tin^ rthe Broject: JDirector prior 
to site visits ;and were debriei-siii npon XHeir irssiiurn. mtan judged the 
consistency of response patterns reported by f -ntra interviewej-s, ^ felt 
sufficiently successful in having overcome tthe first prabiieta.. 

Analyzing the result,^ presented a mnore ..tlliif^ For various 

reasons, it was impossibDM to qioismtdiey t-iie rta-spnnses. By- d^gn, our in- 
terview guides did not pemrrt tiris t^e ox^an^ysis. Cer.i:E^n questions 
were not- appropriate f or lall rcspondient :.gr^ or indxvidiusis within the 
groups. Some people had net used tine pmgran at: all or at li^st suffi- 
ciently to respond to cent^ain questxcms., Our tallies would inave been re- 
plete with »»Not approprlaite:" for the^e .iB-spoimiients . Since ^ome teachers 
had to be interviewed in sro^ups., we re acCTHd: -several c:om3ns±tie viewpoints, 
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whicii may or may not have reflected accurately each individual opinion 
within the group. 

Instead, we based our qualitative analysis of the interview data on 
the procedures described by Lofland (1971, Chapter 6). Notes were devel- 
oped from the initial interview records and were filed in various ways: 
by topical issues, by respondent positions , and by site locations. Coiranon- 
alities were culled from the data; inconsistencies were noted and studied; 
txends were abstracted. 

The interview results reported in subsequent chapters are based on 
these methods of examination, re-examination, and abstraction. We have 
not :i^orded individual interview data. Neither do we see any advantage 
in recording unique situations, nor do we want to jepordize the prospects 
for productive in terviews ' wi th some of the same respondents or their 
colleagues in the future. 

Program Monitoring 

A mailed questionnaire was used to obtain information from teachers 
about how they used USMES in their classrooms* (These end-of-the-year , 
summary viewpoints complemented other data that were also used to assess 

actual program use.) 

The Program Monitoring Form, shown in Appendix C, consists of two parts; 
(1) a series of 11 open-ended questions, and (2) a rating scale. The eval- 
uation team developed this form after we interviewed 80 USMES teachers 
during our site visitations. We based these questions on what we per- 
ceived to be widespread concerns among the teachers about introducing a 
unit, sustaining students » interest, and anticipating those preparations 
and resources needed for a unit. Since time constraints on the teachers' 
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schedules had forced us to limit the number of questions we could ask 
during the interviews, we hoped that this use of the Program Monitoring 
Form would enable us to: (1) gather more extended and representative 
information on how USMES units were being used in the classroom; and 
' (2) corroborate or*verify teacher interview responses about the effects 
•of their USMES units on their students' behaviors. 

In May, 1973, the Program Monitoring Form was sent to 105 USMES 
development and implementation teachers who had used at least one USMES 
unit during the 1973-74 academic year. An accompanying memo from the 
evaluation team explained the purpose of the form and notified the teachers^ 
that they would be paid for its completion. Second notices and duplicate 
forms were sent to non-respondents one month later. The final return rate 
was 83/105 or 79%. The respondents included teachers from California, 
Colorado, Georgia, Iowa, Massachusetts, Michigan, New Hampshire, Oregon, 
and Washington, D.C. Collectively, these responses were based on 24 dif- 
ferent ''challenges" or units. 

The questions on Part One of the Program Monitoring Form were open- 
ended and did not lend themselves to ready quantification. Therefore, 
our analyses of these results were based on the processes of content analysis 
review, abstraction and synthesis. In addition, we used anecdotal informa- 
tion from these forms to illustrate the trends, or more often, the variety of 
positions which we culled from the data. 

Part Two of the Program Monitoring Form consisted of a scale requesting 
teachers to rate the emphases which their units placed on various content 
areas and learning activities. Since fully 24 different units were repre- 
sented by the respondents, the ratings were collated by activity across all 
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units. The data are presented simply as rank ordered percentages of 
respondents who felt their units emphasized each given content or activ- 
ity. 

Other Data Collection by Trained Observers 

During a three day period in August, 1973, observers were trained 
to administer the Stanford Achievement Tests and the Playground Problem, 
and also to use an observation scale which would enable the evaluation 
project to have a more objective accounting of the classroom activity. 
This observation technique and the. results of its application are re- 
ported in Chapter 111. The Classroom Activity Analysis Form itself is 
illustrated in Appendix D. 

Near the end of the school year, the observers were asked to complete 
a School Information Form for each sample school, and a Class Information 
Form for each teacher in the evaluation sample, both USMES and control. 
The forms are shown in Appendices A and B. They were designed to ob- _ 
tain descriptive and classification information to characterize the sample 
schools and classes and to be used as independent variables in other analyses. 

Chapter Summary 

This chapter detailed the evaluation design of this project, partic- 
ularly its bases of sample selection, the instruments employed for its 
data collection, its methodologies of data collection, and the techniques 
of analysis applied to the resulting data. 

Purposive sampling of the four designations of USMES teachers was used 
to achieve a sample" of USMES classes representing a cross section of USMES- 
teacher-experience, unit challenges, grade levels, socio-economic levels 
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and geographic areas. Contnrol teachers at the same grade levels, and from 
the .^ame schools w^ere selecitied only for USMES implementation and Chicago 
classes. Sample .aittrit:ion\(2ne to late funding and other problems was 
serious. Actual data is Based on 19 schools with a variety of population 
settings and socio-ex^aaonric levels. 

Class Infonira:tlcm:-:5'onms, soliciting descriptive and classification data, 
were obtained troTH li USMES development teachers and 13 implementation 
teachers, plus 12 mox-e from control teachers. The teaching experience, 
past training in mathi/science , and class size were noted for each category. 

Data coll ectinHnu efforts were expanded by site visitations, interviews, 
•and teacher queslrLcmnafrss. A total of 80 teachers and 38 administrators 
were involved in ori-sfrire oinsliruG tared interviews by the evaluation staff 
between March and May, 1974. A Program Monitoring Form was sent to the 
orig&al sampLe of USME'S teachers, but only 28 had sufficiently used USMES 
to be capable of rsspondiiig informatively. The questionnaire was then sent 
to 77 others. 

These 8:0 teacheinrs repxresented a cross section of new and experienced 
USMES teachers at all graaie levels. Only 36 were interviewed individually 
by -a two-person :team, and. the remainder were interviewed in small groups. 

The interview data were analyzed by qualitative methods. Teacher in- 
terviews were f^iillowed up by a mailed Program Monitoring Form to gather 
more extensive and representative information on USMES use in the classroom 
and to verify the teacher interview responses about the effects of USMES 
on student behaviors*. 

To test the problem solving ability claimed by the USMES program, 
irrained observers administered the specially developed Playground Problem. 
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The program's effect on the students' acquisition of basic skills, involved 
two sub- tests of the Stanford Achievement Test, Arithmetic Computation and 
Reading Comprehension. An observational Classroom Activity Analysis Form 
was also administered. The interpretation of this acquired data will be 
presented in subsequent chapters. 
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CHAPTER III 
A DESCRIPTION OF ACTUAL USMES USAGE 



The Need for a Description of Actual Use 

Meaningful evaluation of an innovative curriculum project like 
Unified Science and Mathematics for Elementary Schools (USMES) should 
include a description of the way the new program is ac tually being 
used in field settings, as well as a descr i ion of the more tradition- 
al treatment groups, or "control" groups, against whose performance 
the success of the USMES-taught classes is measured. 

Charters and Jones (1973) noted that many evaluators expend con- 
siderable resources developing appropriate outcome measures and planning 
and executing elaborate research designs without attending to crucial 
description of the actual differences between the programs in the "ex- 
perimental" and "control" situations. Writing in the Educational Re- 
searcher, Charters and Jones (1973) underscored the importance of such 
documentation for meaningful program evaluation: 

"What is not standard practice in evaluation 
studies is to describe, let alone to measure, 
how the programs in "experimental" and "control" 
situations actually differ from one another - or 
even to certify that they do.. There are certain- 
ly circumstances in which differences between 
what researchers regard as "experimental" and 
"control" programs are more fictional than fac- 
tual, but in the absence of a measurement tech- 
nology or tradition, such circumstances may well 
go undetected. Then the researchers' findings 
of no consistent differences in student otucomes 
between "experimental" and "control" programs 
can fundamentally mislead educators regarding the 
substantive worth of innovations." (Charters & 
Jones, 1973, p. 5. ) 

This descriptive component is especially important for an evaluation 

of USMES. Unlike more structured curricula which might prescribe relatively 

i 
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uniform student and/or teacher activities through texts, workbooks, 
teacher guides, programmed instruction, etc., USMES is purported to be 
"an important new style of education" (USMES Central Staff, March, 1973, 
p. 1)' designed to involve students in real problem solving. While a 
series of challenge units and tangible resource materials have been 
developed by USMES, this program, according to its developers, is more 
accurately portrayed as a- philosophy of education than as a collection 
of materials. Each USMES challenge unit should evolve from the children's 
identification of, and action on, a problem which is real and important 
to them. By design, the USMES approach could result in as many different 
treatment groups as there are classes using USMES. 

The purpose of this chapter is to describe how USMES classes actually 
used this program, to differentiate the treatment USMES classes received 
with the math/science programs used in control classes, and to distinguish 
between the classroom activity patterns of USMES and those of the control 
classes. The primary bases for these descriptions are data from the follow 
ing forms: (a) School Information Forms; (b) Class Information Forms; 
■(c) USMES Program Monitoring Forms; and (d) Classroom Activity Analysis 
Forms used for both USMES and control classes. These forms are exhibited 
in Appendices A, B, C, and D, respectively. Also included is pertinent 
information gathered from teacher interviews. The interview guide is shown 
in Appendix G. 

Length and Intensity of USMES Usage Experiences by Sample USME S Classes 
A. Development Classes 

First will be summarized the answers given by the 15 sample development 
teachers to the series of questions (1-6) on the Class Information Forms. 
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These deal with the teachers* and students' experience with USMES. Ten 
of these development teachers had been with the program for two years 
and five more were completing their third year. Over this two- and 
three-year period, these teachers had used between one and five units, 
in addition to the development units assigned for 1973-74. The list of 
units they had used virtually exi. iasts the units available for implemen- 
tation by 1973. Their 1973-74 development units were • distributed as 
follows: 

Number of Sample Teachers 
Name of Development Unit Working on that Unit 

Advertising ^ 

Animal Behavior 2 

Bicycle Transportation 1 

Classroom Design ^ 1 

Community Services 2 

Ecosystems 2 

Learning Processes 2 

Manufacturing 3 

Music Production ^ 

Many persons involved in or interested in the USMES development and 
this evaluation project have pointed to the long term effects of the pro- 
gram as a very important issue for investigation. In preliminary re- 
sponse, we included questions on the Class Information Form about the 
students' experience wich USMES. We wanted to corroborate other data on 
how intensive and extensive was the application of the treatment we were 
evaluating. 

As to the length of exposure to USMES, only four teachers had classes 
with children who had prior USMES experience. The percentages of these 
USMES-experienced children in those four classes ranged from 20% to 60%. 
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As to the intensity of USMES usage during 1973-74, the pattern which 
emerged varied. An inspection of Table 2 will reveal that the development 
teachers spent anywhere from 1 to 4 hours per week on USMES over a 6\ to 
36 week period. 

Three of the 15 development teachers expressed no interest in using 
USMES the following year. We did not pursue the reasons for this negative 
response. It may have be6n disenchantment with the program; it may have 
been for other reasons. 
B. Implementation Classes 

By definition, implementation teachers were less experienced with USMES 
than the development teachers described above--all of the implementation 
teachers were using USMES for the first time that year. The units which 
they were using were distributed as follows: 

Number of Sample Teachers 
Name of Implementation Unit Working on >that Unit 

Consumer Research ^ 
Describing People - 3 

Designing for Human Proportions 4 
Dice Design 1 
Traffic Flow 1 
Weather Prediction 3 

While these implementation teachers were themselves inexperienced with 
USMES prior to 1973-74, two of the 13 sample implementation teachers had 
students with prior USMES experience. In one class, half of the students 
had used USMES before; in the other class, the figure was 80%. 

The picture of the intensity of 73-74 USMES usage, emerging for the 13 
sample implementation teachers, was exceedingly varied. Seven teachers hac 
used only one unit during that year; four ujed two units, and two teachers 
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TABLE 2 



Time Spent on USMES Activities Including Design 
Lab Work, by Fif teen Sample Development 
Teachers During 1973-74 



Hours/Week Weeks/Year Total Number 

Teacher Spent on USMES Spent on USMES of Hours 



A 


3 


19 


19 


B 


7 


23 


? 


C 


1 


12 


12 


D 


4 


23 


92 


E ' 




23 


? 


F 


3 


36 


108 


G 


3.5 


28 


98 


m 


4 


24 


96 


I 


3 


30 




J 


3 


30 


90 


K - 


3 


26 


78 


L 


2.5 


30 


75 


M 


3 


6.5 


19.5 


N 


2 


28 


56 


0 


.75 


30 


22.5 



Average (based on N=13) Mean - 68. 

Median = 78 
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had used three. However, the number of units alone can not be used as 
a yardstick to measure intensity. The variability in time spent on USMES 
by the implementation teachers can be seen in Table 3. This data under- 
scores the difficulty which one faces in trying to characterize the nature 
of the treatment, i.e., the USMES curriculum as it is actually applied in 
classrooms. Looking at the factor of time alone, one notes thaX USMES 
might have been used intensively over a :3iiorr. period of time, (^.g. , 
teacher- C)-, in small doses over a great number of wraks (e.g., teacher 
J), or- wi-tia any combination of values of; intensity and duration. 

AEsDXher descriptive note on these L3 sample implementation tteachers 
is that all but one expected to continue using USMES the following year. 

^ We had" intended to describe the ways' 'that ]USMES teachers in ■self-con^- 

tained classrooms reportedly apportioned their instruction time among the 
various subjects when USMES was and was not being used. However, we error- 
ed by omitting the column headings, "When USMES is used," and "When USMES 
is not used," over the two columns of lines on page 2 of the Class In- 
formation Form. This error precluded a meaningful interpretation of the 
time periods reportedly spent by USMES teachers on the nine areas listed 
in question 2, and page 2. We could not say whether the teachers had in- 
cluded or excluded USMES in their assignments of the time spent on math, 
science, social studies, etc. Unfortunately, this error also prevented us 
from making comparisons between USMES implementation teachers and their 
controls on this distribution of instruction time. 
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TABLE 3 



Time Spent on USMES Activites, Including Design Lab 
Work, by Thirteen Sample Implementation 
Teachers During 1973-74 



Te acker 



Hours/Week 
Spent on USMES- 



Weeks/Year 
Spent on USMES 



Total Number 
of Hours 



A 


1.5 


7 


? . 


B 


1.5 


20 


30 


C 


6 


2.5 


15 


D 


3 


• 25 


75 


E 


2.5 


38 


95 


F 


1 


12 


12 


G 


1 


8 


8 




.6 ■ 




; 96 


I 


3 


12 


36 


J 


2 


26 


52 


K 


5 


7 


35 


L 


5 


13 


65 


M 


2 


20 


40 



Average (based on N=12) 



Mean =46.58 
Median = 38. 
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Non-USMES Curricula in U,SHES and Con.trol Classes 
A. Development Classes 

When a'sked on the Class Information Form to list the names of the non- 
USMES science, social studies, math and language arts programs and texts 
used by their classes, the: USMES development teachers responded with a 
lengthy set of replies defying anything but gross categorization. The USMES 
development teachers used, in addition to USMES, only texts, no texts, or 
most frequently, a combination of texts, self -developed materials, com- 
mercially prepared workbonks, and parts of other science/math curriculum 
programs including ESS, SCIS, SAPA, ISCS, and IPI. 
B. Implementation Classes and Their Controls 

It should be emphasized that the control classes for the 1973-74 USMES 
evaluation were selected from the same schools and at the same grade levels 
at their USMES counterparts in the sample. We did this to minimize the 
differences between USMES and control classes in these extraneous factors 
which could account for differences in program outcomes. Thus, it should 
not be surprising that there was great consistency. between USMES and control 
classes within a school in the kinds of non-USMES curricula which the 
teachers reportedly uised. Across schools, one noted an enormous variety 
of texts, programs, and materials used for non-USMES science, social studies, 
math and language arts, while, within a school, USMES and control teachers 
tended to use the same programs, texts, and materials outside of USMES. 

The Influence of USMES on Control Classes " 

While the selection of control classes from the same schools as the 
experimental classes had the advantage of minimizing extraneous variance, 
this practice also had the undesired effect of reducing the treatment 
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differences between USMES and control classes. The "contaiEriration" of 
non-USMES classes with the USMES program was a phenomenon (Served by the 
members of the evaluation team during our site visitations earlier in the 
year. For example, one teacher, who had not been trained ifi. the use of 
USMES, borrowed ideas and materials .from an USMES- trained: ^cdLleague and 
effectively persued an USMES challenge in her classroom. To- label this 
kind of teacher, as a "control teacher" would have been very i.iisleading. 

Furthermore, some students who had used USMES prior to the 1973-74 
evaluation year were now assigned to classes taught by teachers not trained 

to use USMES, i.e., to potential control teachers. To use as control data, 

performance data based on these students w.iiuh fsriox USffilS experience would 

have been equally misleadrng. 

Observers were directed tffl ask control teachers the quiestions on- page 
three of the Class Information Enrm in an e£f:art to assess drfee possible 
contamination of control classes by exposure to USMES. One of the 13- 
control classes might better have been classified as an USMES class be- 
cause that teacher claimed to have: used the USMES .pMlosophy im: her classes 
and to have read some unit resource books for tEuchers, although. she did 
not use an USMES-developed unit. Moreover, her students had used USMES in 
prior years and were continuing to use the design lab facilities. Two 
other "control" teachers of the group of 13 claimed to know and to sub- 
scribe to the USMES philosophy. While they say they use the USMES' philos- 
ophy in their classes, they are not familiar with any USMES-produced material 
nor had their students been exposed to USMES. The remaining 10 control 
teachers claimed to have no knowledge of the USMES program, nor its philos- 
ophy, materials, or approaches. Data only from the latter 10 control 
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classes could be used as control data in the analysis of student per- 
formance. 

Design Lab Facilities in the -Schools 

Of the 19 sample schools where USMES was being used, 6 were in their 
first year of USMES usage, 8 in their second, and 5 in their-third. Fif- 
teen of the 19 sample schools had design labs; 4 did not although one of 
these 4 was a middle school with an extensive industrial arts workshop 
available for use by USMES classes. 

On CTiar site visits, USMES teachers expressed concern about being able 
to use a well equipped, adequately staffed design lab during their USMES 
units. For this reason, we included questions about the design lab facil- 
ities, materials, and staffing on the School Information Form. The number 
of hours per week that each school's design lab was staffed by someone 
other than the classroom teacher using USMES were distributed as follows: 

Hours During Which Number of 

The Design Lab is Staffed Schools Reporting 

0 7 

8-11 3 

15 2 . 

20 1 

30 J: 

14 

The staffers included released time teachers, paid teacher aides, adult 
volunteers, building principals, and a retired Army Sergeant. None of 
the 15 sample schools used older students to supervise younger students. 
(One of the 15 schools had one design lab as a permanent part of an USMES 
teacher's classroom; for them, this question was inappropriate.) 

The quantity of tools contained in the 15 schools' design labs were 
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appraised as follows: 

Tools Available in Design Lab Number of Schools Reporting 

Very few tools 1 

Basic tools 5 

Extensive tools 7 

Not answered ^ 

15 

Nine of the 15 sample schools reported they had designated a separate 
room for the design lab facility. The remaining schools were pressed for 
space and had to resort to other arrangements for their labs. The follow- 
ing situations were reported: 

Location of Design Lab Number of Schools Reporting 

Separate room ^ 

Permanent part of classroom ^ 

Portable design lab cart^ 3 

Part of auditorium ^ 

Off school grounds in another building ^ 

15 

Teachers' Viewpoints on the Place of USMES in the School's Curriculum 

This assessment is based on the interview responses of 80 USMES teachers 
The questions dealing with the place of USMES in the school's curriculum are 
found on, page one of the Interview Form for USMES Teachers (Cf, Appendix G) 
The sample of teacher respondents is described above in Chapter II. 

We opened the interviews by asking the teacher if he considered USMES 
a supplement to, or a replacement for, the subjects of math, science, and 
social studies, A small minority responded saying that, if one followed 
the philosophy completely, USMES would be a replacement for these subjects,^ 
It would require careful planning, but, as one teacher put it: "Anything 
can be taught under the "umbrella' of a unit." However, the vast majority, 
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of USMES teachers interviewed saw USMES as a supplement to regular class 
work, especially in math and social studies. They felt that math skills 
in particular needed to be taught directly and then could be reinforced 
by the 'p^^otilem solving activities of USMES. 

More commonly, USMES was considered a replacement for science, possibly 
because it was the only science which many teachers taught. Many teachers 
used USMES to satisfy the amount of time for science teaching required by 
their districts. If the requirement was an hour-a-day for five weeks, 
and on plantk, primals or the solar system— and such was the case in sonie 
districts we visited— then the teachers wanted an USMES unit tailored to 
those specifications. 

No teacher viewed USMES as a replacement for his regular social studies 
instruction. Although the teachers' perceptions of the program were gov- 
erned by the unit they were using— the Burglar Alarm unit was a "science" 
unit; any unit containing graph work was a "math" unit-few teachers seemed 
to label any unit a "social studies" unit. 

Some of the teachers who used USMES as 3 supplement to these three con- 
tent areas said they were doing so because of the demands of their districts 
to "cover" certain content areas during the school year. These teachers 
felt that concentrating too much effort on USMES might hinder their ability 
to meet these demands. Other USMES teachers who used this program on a 
limited basis, cited demands from parents: that teachers cover prescribed 
content; that students do well on standardized tests; that their students 
compare favorably on traditional criteria to other children at the same 
grade level. Obviously, these teachers were unwilling to risk giving up 
what has worked for them in the past in their attempts to satisfy the demands 
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" real or perceived--of their districts or of the parents of their students. 

In answer to another question as to whether USMES really is an integrated/ 
approach to the teaching of math, science anti social studies, the over- 
whelming response from teachers was ''yes." Only one person issued a ^i^g- 
ative response. Several teachers noted that it depended on the unit. It 
is interesting to note that, although the teachers considered USMES an inter- 
grated approach, they had no difficulty separating out the various content 
areas. 

Results from the Program Monitoring Forms; How Were USMES Te achers Using 
the Program in Their Classes? 

The Program Monitoring Forms shown in Appendix C were completed and 
returned by 79% (83/105) of those USMES development and implementation , 
•teachers reported to have used at least one USMES unit during the 1973-7^ 
academic year. Collectively^ their responses were based on 24 different 
units. The results are summarized below, question by question. 

Question 1, 2, and 3 dealt with issues of vital concern to teachers, 

issues expressed to us repeatedly during our interviews with them the pre-: 

vious Spring. They are: how do you introduce a challenge; and once it 

is introduced, how do you set realistic and meaningful goals? 

QuestiLon 1: How was the unit you are presently 

working on introduced to the students? 

After studying the responses to this question ' from 83 questionnaires, 

it seemed obvious to us that only rarely did any of the challenges arise 

initially out of concerns expressed by the children. Rather, as in most 

classroom teaching, the USMES teachers either contrived situations and 

hoped that, as a result, the problem would arise, or they posed the problem 
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to the children directly. The ways in which the situations were "set-up" 

varied so greatly that it was impossible for us to draw generalizations 

without presenting a distorted picture. 

Question 2: What were some of the typical student 
reactions to the unit? 

Generally, student reactions to the unit were reported to be very favor- 
able. Excitement, enthusiasm, and enjoyment were reactions noted by many 
teachers. In responding to Question 2, only a few teachers cited boredom, 
confusion, or disinterest. Some of the more detailed responses noted a 
variation of student reactions over the course of the unit and/or between 
one child and another. 

Question 3a: What were your goals for this unit? 

Question 3bj How did the students define the challenge 
for their situation? 

Extreme variability across responses was quite evident in the teachers' 
answers to this goal setting question. Indeed, considerable diversity in 
responses to Questions 3a and 3b was expected among teachers who were using 
different units. However, some teachers identified general goals; others 
cited specific behavioral objectives. Many answers did not really address 
the questions. Perhaps some of the teacher respondents were not sure of thei: 
goals or their students' goals for the unit; instead they responded to Ques- 
tion 3a and 3b by listing the teaching/learning activites or by elaborating 
on their responses to Question 2. 

To afford the reader a picture of the broad scope of the methods which 
the teachers used to present the unit and of the wide range of student reac- 
tions to the challenge, we have chosen to present a cross-section of the tea- 
chers responses to Questions 1, 2, and 3. We further decided not to take a 
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random sample of responses, but to deliberately select answers which illus- 
trate the variation in content, length, and depth of teacher responses 
within one unit (Describing People), across several units, and across 
grade levels. In addition, we decided to present selected answers for 
Questions 1, 2, and 3 from the same teacher, to give some notion of how 
the goals may or may not relate to the method of introducing the challenge. 
The sample responses are shown in Table 4. 

In reviewing the teachers' responses to the 11 open-ended questions 
on the Program Monitoring Form, we noted with interest that no differences 
in the teacher responses could be attributed to either grade level or unit 
factors. (The only exception to this observation appeared in the responses 
to Question 9: From the teachers' view, the content emphases on math, 
science and social science and the resulting attitude changes in the children 
toward these subjects were unit-dependent.) Further study of the answers 
to Questions 4 through 11 on the Program Monitoring Form suggested that it 
was two other factors which accounted for some of the differences in teacher 
responses: frequency of USMES usage, and teacher directiveness. 

Question 4a: Did the children lose sight of the 
goals during the unit? 

Question 4'b; If yes, ti. why in your opinion did 
this occur? 

Question 4 was the only item on the Program Monitoring iForm which e- 
licited consistent responses from the teachers. The overwhelming majority 
felt that the students did lose sigtit of the goals during the unit. Re- 
sponses did not differ by unit or by grade level. Both lower and upper 
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Selected Responses From USMES Teachers to Questions 
- From the Program Monitoring Form 



Unit 



(Question li How was 
the unit you are pre- 
sently working on in- 
troduced to the stu- 
Grade dents? 



Question 2 ; What; were 
some of the typical 
student reactions to 
the unit? 



(Question 3a ; What were Question 3b ; How did the 
your goals for, this students define the chaU 
wit? lenge for their situation? 



Describing 
people 



1-2 



I Introduced describ- 


The children enjoyed 


They defined their particu- 


ing people by motiva- 


the unit tremendous- 


lar challenge as an^ inter- 


ting the children to 


ly. They were fas- 


esting) workablei and a 


develop games using 


cinated by the ways 


problem solving technique 


descriptive ternis« 


and terms in which 


which they hoped to master 




each of them, as well ' 


by themselves. 




as others could be 




described. The chil- 






dren wanted to make 






graphs as a result 






'of some of their 






findings. 





ERIC 



He were talking about 
t.V. shows and es- 
pecially police stor» 
ies. They mentioned 
a local store being 
held up and how the 
police caught the 
robbers because the 
owner identified and 
described them. 



Some of the identify- 
ing factors they 
brought up were hil- 
arious. They had 
small arguments over 
which parts of a de- 
scription were best 
and would help the 
most in identification. 
Being third graders 
they thought it fun 
to play police. They 
were really excellent 
in their choice of de- 
scriptive words. 



To teach the children 
to be more observant. 
To notice more in the 
environment. To in- 
crease their vocabu- 
lary. To make them 
more aware of other 
persons^ places and 
things about them. 



At first, they considered 
it a game. But as we pro*, 
gressed Into the unit they 
, could see that they all a- 
greed that it was something 
they could do themselves . 
and that sometime it might 
turn to their advantage to 
be able to recognize someone 
or some object. We more or 
less stayed with describing 
people more than anything 
else. 



TABLE 4 (Cont.) 



Unit Grade 



Question 1 ; Hov was 
the unit you are pre- 
sently working on in- 
troduced to the stu- 
dents? 



Question 2: What were 
some of the typical 
student reactions to 
the unit? 



(jtiesti qn 3 a; What were 
your goals for this 
unit? 



Question 3b ! How did the 
students define the chal- 
lenge? 



Describing 
people 



4.5 Asked one student to 
go to another room and 
bring back a blue 
eyed, brown haired 
person. Imraediate re- 
• sponse was. "Vou 
didn't tell us enough," 
Started small groups 
to write descriptions 
of the person I had 

sent for. 



Describing 
people 



Describing 
people 
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Ranged from boredom 
at times to high ex- 
citement at others. 



To reach unit chal- 
lenge to introduce 
USMES appvoach to the 
children. To have 
children experience 
"learning" as opposed 
to traditional educa- 
tion. Have children 
experience problem 
solving situations. 



To describe someone using 
a list of good descriptions 
and the fewest (descrip- 
tions). 



We had been working on 
adjectives by listing 
characteristics of 
people. Played sit 
down game with charac- 
teristics chosen to 
describe a person 



Excitement. 



To present a real life 
problem and finding ways 
to solve problems. 



"That's easy. 



We were working on a 
unit on the human body. 
One of the counselors 
came in on the pre- 
tense of finding out 
some' information. 
When she left I had 
the children to de- 
scribe her, She re- 
turned later and had 
changed clothes, We 
discussed the fact 
that we all tend to 
describe clothing 



The children were enthu- 
siastic and made com- 
ments to the effect 
that they couldn't' wait 
for Thursday and Fri- 
day to come, (USMES 
days). 



My main goal was for the 
children to become more 
observant of people and 
their individual charac- 
teristics. 



The students worked to ob- 
tain the 5 characteristics 
bhey deemed to be th very 
best characteristics with 
which to describe a person 
When they established thes 
5 characteristics they the. 
went on to clarify exactly 
what they meant by each one. 
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Unit Grade 



Question 1 ; How was 
the unit you are pre- 
sently working on in- 
troduced to the stu- 
dents? 



[juestion V , What were 
some, of the typical . 
'student reactions to 
the unit? 



Question 3a ! What were Question 3b ! How did the 
your goals for this unit? students define the chal- 
lenge? 



learning 



5 It was introduced after 
a test, by asking how 
could they have done 
better on the test-- 
this led to the dis- 
. cussion of different 
ways of learning. 



Mays of 
learning 



Learning 8 
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Some were very confused-' To have each child sat- 
soitie very entliusiastic isfied that he had coin- 
about testing each other, pleted his testing and 

had some results that 
had a value to him. 



Most of them got learning 
and teaching concepts con- 
fused. Their learning In- 
volved different methods 
of presenting materials 
like teaching. 



8 The students have been 
exposed to many ways of 
learning (assumed) this 
year. The teacher and 
the students thought 
that it would be nice 
to explore tne posbibil- 
ity of learning content 
by means of creative 
problem-solving. 



The unit was begun dur- 
ing the last two weeks 
of the school year. The 
students are very ex- ' 
cited about learning 
content and having fun 
in the meantime. 



To deliberate techniques 
for the production of 
new ideas and idea-com- 
binations. 

To produce new ideas con- 
sciously and fieliberate- 
ly without waiting for an 
unpredictable inspiration. 



How can we learn content 
and still 'be creative. 



The teacher asked the 
students if they had 
ever studied percent. 
Since so few students 
had, the teacher asked 
the class how they 
thought they could 
best learn about per- 
cent. The discussion 
then went to a more 
general one about how 
they learned anything .. 
best. The class de- ^ 
cided to run an experi- 
.rnpnf .ahoiit learnin2. 



Very competitive, 
highly motivated to 
learn about percent, 
Very interested in 
how they learn- best, 
more task oriented 
than, with previous 
units. (These are 
low ability classes.) 



To have these low abil- 
ity students analyze 
their learning process 
and to apply wh;,iL they 
found out about how ■ 
they learned in future 
situations. To teach 
the basic concepts of 
percent. 



After the more general dis* 
cussion on learning, the 
experiment was conducted on 
learning the concepts of 
percent. They identified 
4 or 5 ways to learn such 
as tapes and films, games, 
learning centers, applica- 
tions. 



I 
I 
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Unit Grade 



Question I ; How was 
the unit you are pre- 
sently working on in- 
troduced to the stu- 
dents? 



Question 2 ; What were 
some of the typical 
student reactions to 
the unit? 



Question 3a ; What were Question 3b ; How did the 
your goals for this unit? students define the chal- 
lenge? 



Growing 
plants 



Growing 
plants 



Nature 
area 
(Nature 
trails) 



Nature 
trail 
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The challenge - Grow- 
ing a planD for their 
Mother's Day gift* 
Discussed different 
things that would be 
nice for their inoth*- 
ers, something that 
their mother could 
keep over a long per- 
iod of time. 



All the students 
really enjoyed work- 
ing with the plants, 
taking care of them 
and watching them 
over the period of 
time. In the end each 
child was so proud of 
the plants. 



For each child to 
really learn the parts 
and their functions of 
a planti Each one had 
to take care of his or 
her plant. But the 
main goal was for each 
child to survive. 



To be a success to grow 
their plants for Mother's 
Day. 



Discussion of plants 
in the classroorn» hav- 
ing a garden at schooL 



Much excitement and in- 
teresti 



Appreciation of plants 
and caring for a school 
garden. 



How to grow plants at school 
school. Bake sale to earn 
money to buy plants (pro- 
ceeds; $93,00), Plant 
sale (proceeds; $55>00)> 



Nature walk through 
woods behind school 
with students record- 
ing observations (they 
used 5 senses and had 
had previous training 
in observing). 



They liked the study 
very much and were ex- 
cited about it. They 
would stdy after school 
or do extra work at 
home, They especially 
liked being outside. 
One student invited the 
class for a walking 
hike to her home to 
study tree identifica- 
tion, terracing, plant 
identification. 



To develop a site to 
serve as a laboratory 
for outdoor investiga- 
tion. To increase the 
students* awareness of 
nature. To function as 
the setting for a cur- 
riculum enrichment pro- 
gram, 



To preserve and to study 
the natural life of an area 
behind the school and to 
make a place where people 
can enjoy themselves and 
learn about nature, 



How could we make a 
place at school to , 
help others and our- 
selves learn about 
nature? 



Generally very enthusi- 
astic - wanted to go 
outside and start look- 
ing for a place immedi- 
ately. 



To make a nature trail. 
To label trees* To make 
■ learning centers along 
the trail - ran out of 
time, " 



We have quite a bit of land 
surrounding the school which 
is forest. They thought of 
a nature trail and wanted to 
explore the woods for the 
best .place. They were eager 
to have a trail and take 

other classes through the 

area and teach about what 
they'd learned, yy.-.,\: 
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grade teachers felt that their students became distracted solving minor 

or tangent problems which took them away from their overall goals. 

'*They tend to keep discovering new problems 
before any are solved and have trouble es- 
tablishing priorities.*' 

''Yes, some of the sub-challenges became more 
exciting than the original problem.'' 

"Thoy got carried away with inventing drinks 
and forgot to find a best drink. Besides, it 
was more fun th-an^-wrestling with the other as- 
pects of the problem." 

When asked why the children lost sight of the goals, many lower grade 

teachers commented that this happened because the children were young, 

and they constantly needed to be reminded of where they were going. 

"Of course they did — by virtue of age (6-7 
year olds), and the nature of the beast." 

"Young age of children. They were more in- 
ter'^sted in experimenting with sound." 

Question 5: How often and in what situations 
was refocusing required during 
the unit? 

In responding to Question 5, most teachers agreed that they had to 
redirect their students' attention toward the goals of the unit, but how 
often they were required to do this depended on the frequency each teacher 
used the USMES unit. If the students worked on the unit daily, refocusing 
was necessary only infrequently; if USMES was used only once a week, re- 
focusing was required at each meeting. 

"When several days elapsed between USMES sessions, 
a general review was needed to refocus students' 
thoughts toward goals and challenges." 

"It would be great if this were a unit used each 
day rather than twice a week." 
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*'Yes, after vacations, camping for a week, 
and other special events which were so time 
consuming that we didn't have time for USMES," 

"Yes, after long vacation lapses." 

Question 6a: Were there fluctuations in stu- 
dent interest during the work 
on the unit? 

Question 6b: If yes, please explain at what 
points these occurred. 

The value which USMES teachers attached to intensive use of USMES was 

certainly one of the strongest issues to 'come out of the Program Monitoring 

Form. In responding to Question 6, many teachers commented on the nec- 

cesity of not spreading an USMES unit over a long period of time. They 

recommended that once a unit is started, it should be done often, not 

merely once a week. 

"More interest was apparent when sessions 
were scheduled closer together." 

"I feel that student enthusiasm would have 
been more favorable had the unit been pre- 
sented in a more concise manner. Time lapses 
between sessions caused a break in continuity 
and lessened student interest." 

Apparently much enthusiasm was generated at the start of a unit, and 
the approaching solution of the problem appeared also to be reinforcing. 
But during lapses of time, whether vacations, or time intervals from "USMES 
day" to "USMES day," the reinforcement declined. This time factor and the 
nature of the activity being pursued were the two most frequent reasons 
given for fluctuations in interest as the unit progressed. 

Interest in USMES was greatest when children were involved in physical 
projects: working with tools; going to other classrooms to collect data; 
going outside the school for nature walks. However, it appeared that once 
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these activities were completed, children lost interest. 

'^Interest lagged when we were not actually 
doing mixing or testing or selling." 

"people who raised plants soon got bored. 
Once containers were built, simply observing 
became much less exciting." 

"(Interest lagged) during more pedestrian 
labors which required time, patience, and 
accuracy, such as measuring." 

"During graphing." . . 

"Children are happier in units that involve 
physical activity. The nitty-gritty or 
writing down and tabulating turns them off." 

"USMES was fun when we built cages, but after 
that it got boring," 

Question 7; If the children hit "any impasses 
during the unit, an what points 
did this occur and how was the 
impasse overcome? 

Some responses to Question 7 were relevant and some were merely 
repetitious of the responses to Questions 4 and 6. Children hit an 
impasse during a unit when they lost sight of the major goal or when they 
completed the physical activity components of a unit. Other teachers 
added that an impasse arose when it appeared that no progress was being 
made, or when groups could not work together, or when the children lacked 
the specific skills needed to continue working toward a solution. These 
comments applied to both lower and upper grades. Those teachers who 
reported a halt in unit progress due to "slow-going," or to lack of stu- 
dent skills usually responded to the impasse by being directive. All 
of these teachers agreed that some additional progress recaptured the 
children's interest in the problem. Some of these teachers commented that, 
in retrospect, they should have been more directive at these points to 
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help the children deal with their frustrations. 

Reports of frustration and comments on the need for more structure 

were by no means universal, however. Some classes, apparently because of 

the make-up of the groups or because of previous experience working in 

groups, did not encounter these really frustrating experiences. It could 

also be that some teachers were more sensitive to the students' approaching 

sense of frustration and were more directive, consciously or unconsciously, 

at these points. One teacher who was aware of his intervention noted: 

"Sometimes the teacher needs to be directive. 
He shouldn't be inhibited by the 'USMES 
approach ' • " 

Question 8: Please explain the nature and ex- 
tent of any student comments or 
criticisms on your present USMES 
unit and/or the USMES approach in 
general* 

Teachers' reports of students' comments or criticisms of USMES were 
solicited in Question 8. There can be no doubt that in the teachers' view, 
USMES generated a high level of excitement and enthusiasm in their stu- 
dents. 

"All comments were positive, since the units 

were student. initiated and student run." . - 

"Students ask for USMES time and think of it 
as a 'fun' activity." 

"Children really love the unit. They es- 
pecially enjoy doing it themselves." 

The instances of a child not liking an USMES unit were rare. Out of 83 

questionnaires, \here were only 4 teachers who said that a small minority 

of the class was uninvolved. They further noted that these same children 

remained uninvolved in most other classroom activities as well. As we 

noted previously in this report, some parts of the unit were reportedly 
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more popular than others. But in the overall, one must conclude that 

the USMES units were popular activities for at least part of the time. 

Question 9: Have you noticed any attitude 
change (+, in the children 
towards math, science, social 
studies, or any other subject 
since the children began work 
on the USMES unit? 

Whether or not this student enthusiasm and excitement with, USMES was 
accompanied by changes in their attitudes toward specific subjects or 
toward school in general was an issue which elicited a variety of teacher 
comments. (We had also asked teachers to respond to this issue during 
our on site interviews. Their comments then were equally diverse.) None- 
theless, some meaningful response categories can be summarized. 

The majority of teachers noted positive changes in student attituoes 

toward specific subjects, but not all subjects* Some teachers expressed 

the view that selective changes occurred when the various units emphasized 

a different subject. Some units, for example, stress math and others 

science. None were seen to emphasize social studies. 

'*The unit afforded a lot of math, and even the 
slower children showed a positive attitude,*' 

.'^ath would be a plus. Not sure on other 
subjects.'* 

•'Yes, particularly in the maintenance of 
computation skills. Students accepted this 
as part of the challenge and not 'math work'." 

"Some of the children hated math when they came 
in the fall. To date, I have the best math 
class I've ever had." 

"Definitely yes. More interest expressed par- 
ticularly in the science area." 

"Social studies teacher notes they are more 
eager to make decisions about things." 
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A few teachers reported that more general attitude changes toward 
school had occurred in the children since they began working on an USMES 
unit. 

»»I have noticed a positive, attitude toward school 
in general." 

About 25% of the teachers saw no change in the children's attitudes, 
either toward specific subjects or toward school in general. 

*'I have noticed no change in attitude at all." 

"NOi They are enthused about the unit, but I 
can't see any carry-over into the other subjects," 

"Their attitude towards these things is good when 
they are working on the USMES project. I've not 
seen that this runs over to their regular math 
classt " 

"Not really." 

The remainder of the responses to Question 9 came from teachers in 
schools which departmentalized. Usually each of these teachers would speak 
to his particular subject, but felt he ::ouldn't speak to others. Again, . 
many of these teachers noted positive changes toward their subjects. Others 
did not speak of attitude change^ 

"I only teach science." 

'H/e have separate classes for these subjects." 

Question IC^ Were there any activities in the 
unit which posed special problems 
for you, the teacher, as a class- 
room manager? Please explain. 

Questions 10 and 11 dealt with different facets of the same topic-- 

special needs or problems of the teacher using USMES. The responses to 

Question 10 were quite consistent. Fully three-quarters of the teachers 
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encountered at least some difficulty managing the children in small groups 

or during individual USMES activities, especially in the Design Lab. 

"It was hard to be with all the groups at the same 
time.*' 

"It was difficult to guide small group activity. 
The children were not independent enough to work 
with only limited guidance." 

"I definitely found the need to have more than 
one adult in the lab because of safety reasons." 

This lart comment was echoed repeatedly by USMES teachers, both in their 
responses to the Program Monitoring Form and during our interviews. 

About 25% of the teachers said they had no special problems of class- 
room management with USMES. There could be many reasons for this: some 
groups of children work better together; some teachers can tolerate more 
noise and confusion; some principals are more understanding of the pro- 
gram's need to operate a "freer" classroom. It is also possible, however , 
that once children and teachers have been through several units, and ex- 
pectations are known, things run more smoothly. In view of the concern 
expressed by such a large number of teachers, the special problems of class 
room management for USMES appears to- be an area in need of further study 
and increased attention at USMES Training Workshops. 

Question 11: Were there any special needs that 

this unit required such as materials, 
teacher preparation, teacher aides, 
etc.? 

Question 11 tended ^to elicit more specific needs. Most teachers re- 
sponded by providing a list of materials needed for their unit: rain 
guage, newspapers, radio, thermometers, cotton fabric, yarn, needles, 
staples, etc. The collective list would consume pages if all items were 
included. 7 1 
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Some ot' the teachers who were using units which emphasized science 
mentioned their need for greater technical background. (They did not 
ask for more papers on the subject; they simply" said they needed more 
background.) Several teachers mentioned the need for a list of resource 
people in the community who could be called upon for help. 

About 207o of the teachers replied that they needed to spend more time 

for preparation when USMES was being used. This increased preparation 

time was needed not only to learn background material for the unit, but 

also to collect tools and materials with which to work. Yet, there was 

no evidence that the increased time for preparation would deter these 

teachers from using the USMES program. 

'*It did take a lot of preparation time! But I 
have learned as much as the children!!" 

The answers to the 11 open-ended questions on the Program Monitoring * 
Form were diverse, and analysis of responses to several questions per- 
mitted little, more than gross categorization. However, several points 
were made with great frequency, and these, in particular, seem worthy 
of added note. 

We think there can be no doubt that despite difficulties, the teachers 
who responded to this questionnaire enjoyed using USMES. This was probably 
due in large part to a factor of self -selection, but none- the-less , they 
have found the program workable, profitable for children, and professionally 
stimulating to themselves. Children also looked forward to »*USMES day.'* 
As we pointed our previously, the children particularly enjoyed the physical 
activities which accompany the program. 

There are several basic problems which emerge from the responses: how 
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to schedule USMES for optimum value; now to manage a whole class of 
children either in or outside the classroom; how to collect the materials 
and information needed for a unit so that teacher's preparation time is 
cut down* .Resolution of these 3 issues would make the program, not only 
manageable for the teachers, but potentially more profitable for the 
children* 

The last page of the Program Monitoring Form consisted of a list of 
activities. Each teacher was asked to check the amount of emphasis given 
that activity by the particular unit on which he was working. (Cf. Appen- 
dix C.) These checks were collated, first by unit, and then by activity. 
We decided to include the analysis by activity, rather than by unit, since 
it gives the reader some idea of the proportion of emphasis placed on a 
particular activity across a variety of USMES units. For example, is '^co- 
operation among peers" an integral part of all USMES units or is it 
specific to one? A glance at the data t.veals that it is an integral part 
of all USMES; 97% of the teachers checked the last 2 columns ("heavy em- 
phasis"). Conversly "memorizing" appears not to be a strong component of 
USMES; across all "challenges" or units, only 14% of the teachers felt 
USMES placed heavy emphasis on that activity. 
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The following is a list of all activities, ranked in order of the 
combined percentages of teacher responses in the last 2 columns (''heavy 
emphasis'')* 

Co-operation among peers 
Responsibility for own actions 
Oral Communication 



Problem Solving 
Measuring 
Making Charts 
Tallying 
Writing 

Addition ~" 

Graphing 

Science 

Subtraction 

Mul tiplication 

Reading 

Competition among peers 

Fractions 

Division 

Using Money 

Geography 

Memorizing 

lliiitory 



97% 
81% 
80% 
69% 
66% 
63% 
61% 
60% * 
58% 
55% 
53% 
51% 
43% 
35% 
34% 
32% 
25% 
24% 
17% 
14% 
5% 



EKLC 
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This list does not present many surprises and most of the responses 
can be readily explained. One response which will need further study is 
the item indicating that only 53% of the teachers feel USMES places a 
heavy emphasis on "Science.*' Perhaps definition of the tenn is needed. 
Or perhaps this confirms a comment we heard several times in our inter- 
views: **USMES teaches the scientific method, but not the content of science*" 
This issue needs clarification, and it should be addressed in next year's 
evaluation. 

Results from the Classroom Activity Analysis Forms; What Student Be>- 
hayiorsWi2re Obs erved most Frequently in USMES Classes and in Control 
Math or Science Classes ? 

A. Expec te d Differences in Learning Activities for USMES Versus Control 
Classes 

The Central Staff notes that the USMES program is "an important new 
style of education" (March, 1974, p. 1). Thii assertion is based in part 
on the premise that teachers and students using USMES engage in very 
different teaching/learning patterns from those activity patterns found 
in non-USMES classrooms. 

The developers contend that, in learning the process of real problem 
solving, "students themselves, not the teacher, must analyze the problem, 
choose the variables that should be investigated, i zjarch out the facts, 
and judge the correctness of the hypotheses and conclusions" (March, 1974, 
p. 2). In the USMES mode of learning, the teacher takes on a new role— 
that of coordinator/collaboiator--rather than the director's role typical- 
ly portrayed by classroom teachers. Thus, USMES students are expected to 
engage in activ(s hands-on, "loarning-by~doing. " The "real problem" which 
the students tackle is supposed to provide a focus for various student 
activities: collecting real data; constructing measuring iiistruments , 
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scale models, and test equipment; trying out suggested imporvements; pre- 
paring reports or summaries of their work; presenting their findings to 
the proper audiences. Furthermore, the developers contend that progress, 
toward a solution to an USMES problem requires the efforts of groups of 
students, not just that of an individual student working along. By com- 
parison, children in control classes would be expected to exhibit more 
passive, structured, teacher-directed, and teacher-dominated behaviors. 
B. Procedures for the Observation of Student Behaviors 

The Classroom Activity Analysis form shown in Appendix D was developed 
by Susan Rogers, a member of the USMES Evaluation Team, to enable assess- 
ment of differences in the patterns of activities for USMES versus con- 
trol classes. The categories on the form represent classes of student 
behaviors which could be observed in an elementary school classroom. The 
form underwent successive revisions and pilot-testing over a period of two 
years in USMES and non-USMES classrooms. 

Observers were trained for their proper use of the form. Upon en- 
tering the classroom, the observer conducted seven rounds of observations. 
Each round could take anywhere from a few seconds, if all the children 
were doing the same activity, to a maximum of five minutes. To insure 
a uniform time sampling procedure, the time period between the start of 
each^^round was set at five minutes. During each round, the observer was 
to look at each child as if taking a snapshot, then tally for each student 
that behavior category on the form which best described what the child was 
doing. Lists of observable student behaviors in each category accompany 
the Classroom Activity Analysis Form in Appendix D. 
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TABLE 5 



Results of the 1973-74 Classroom Activity Analysis: .P"""^^^" 

Tallies in 28 Student Behavior Categories During Fall, Winter, and 
Spring Observation Periods for USMES Control Classes 



Observation Period 




Treatment Group. 
Number of Classes 
Category of Student Behavior 



Percentages of Tallies in Each Category 



7. 
8. 
9. 

10. 

11. 

12. 

13. 

14. 

15. 

16. 

17. 

18. 

19. 
20. 
21. 
22. 

23. 
24. 
25. 
26. 



1. Measures 

2. Counts 

3. Constructs 

4. Assembles 

5. Tests/Experiments 

6. Calculates 
Records Data 
Writes/ II lust rates 
Writes (pre -structured) 
Reads How-To-Cards 
Reads-Task 

Free Reading, Writing, Drawing 
Messes Around with Materials 
Talks to Another-Task 
Talks to Another-Social 
Small Group-Task 
Small Group-Social 
Gives Pre-structured Info to 
Teacher 

Gives Original Info to Teacher 

Seeks Info from Teacher 

Talks to Teacher, Social 

Takes ^iirt in Class Discussion 
Presentation 

Listen/Look at Child 

Listen/Look at Small Group 

Listen/Look at Class 

Listen/Look at Teacher 

27. Collecting Material/Maintenance 

28. Resting/Waiting/Fooling Around 



Q Total Percentages 
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1.2 


0.0 1 


1.8 


2.0 


0.0 1 


2.5 


0.0 


0.0 ' 


0.4 1 


1.2 


0.9 


0.0 1 


0.9 


0.0 


7.9 . 


0.4 


11.6 


0.0 


0.0 1 


2.8 


0.0 


4.5 


0.0 1 


3.0 


OoO 


0.0 1 


1.8 


0.0 


18.4 


0.8 1 


1.3 


2.1 


5.7 1 


4.0 


0.0 


8.2 


11.7 


0.9 


1.9 


10.6 1 


. 0.6 


12.7 


6.2 


1.8 


1.1 


6. 5 


0.2 


2.3 


0.0 


0.2 


0.2 


5.6 


2.5 


4.1 


3.9 


0.0 


0.4 


5.7 ' 


2.1 


0. 0 


6.1 1 


0.0 


6.3 


0.2 


0.0 


0.8 


0. 0 


0.0 


0.1 


0.0 


0.1 


1.9 


2.8 


4.7 


0.8 


4.3 


0.0 


0.5. 


0.9 


3.4 


2. 2 


1.2 


1.2 


0.6 


1.1 


1.2 




0. 1 


0.9 


1 ^'^ 


0.0 


2.2 


1.0 


2.8 


3. 2 


3.5 


1 4.6 


3.9 


3.8 


4.7 


1 1.5 


3. 2 


7.2 


4. 1 


8.0 


2.3 


0.3 


1 12.4 


0. 0 


1.4 


1 ^'^ 


0.0 


1.1 


1.9 


0.1 


0. 0 


0.0 


1 0.0 


0.0 


0.6 


1.5 


1 0.5 


1.0 


2.3 


1 1.8 


• 9.4 



3.3 
2.9 
0.2 
4.9 

7.3 
1.4 
2.3 
14.1 
2.6 
2.2 



0.8 
3.: 
0.4 
11.2 

1.9 
7.2 
2.6 
31.0 
4.6 
2.6 



iOO.l 100.2 

7fi 



1.3 
2.0 
0.3 
6.7 

4.8 
1.2 
7.2 
13.2 
4.2 
4.9 



2. 2 

1. 7 ' 

2. 3 
17. 1 

13. 2 
2. 1 
3.2 

21.0 
0.7 
6.2 



■t. 7 
2.7 
0.0 
4.4 

4.2 
1.2 
0.4 
30.1 
2.0 
9.1 



100.0 100.0 99.8 



6.1 • 
2.6 
0.1 
8.7 

2.7 
0.3 
0.5 

28.4 
2.8 

10.3 



0.0 

3.2 ; 

0.0 

10.7 

2.3 
0.0 
1.9 

26.3 
3.5 

U.l 



100.2 99.9 
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(b) • Next, for each record set containing the tallies of an 

observer for one visit to one classroom, the frequencies 
were averaged across the seven rounds. 

(c) If more than one observation was done in a classroom 

* during the Fall, Winter, or Spring, one set in each of 

these periods was randomly selected to use in the analysis, 

(d) For each group (USMES Imp cementation. Control, and USMES 
Development) during ^ach observation period, the behavior 
frequencies averaged across rounds were then averaged for 
groups within periods. 

(e) These average frequencies were expressed as percentages 

of total frequencies for each group in each seasonal period. 
C. Results of Classroom Activity Analysis 

Table 5 presents these relative frequencies of student behaviors for the 
USMES Implementation and control classes during the Fall, Winter and Spring 
observation periods, and for the USMES Development classes which were ob- 
served only during the Winter* The results for USMES classes tend to indi- 
cate that: 

(a) In the Fail, presumably at the beginning of an USMES unit, 
much class time was spent on constructing, assembling, and 
especially on testing/experimenting, but also on calcula- 
ting and recording data. These ^'hands-on»» activities, which 
were related to preparation for, and engagement in, the 
data collection process, consumed almost half (457p) of the 
time during which -0 sample USMES Implementation classes 
were observed. The percentage of time spent on these 

79 



-64- 



activities by USMES Implementation classes tended to 
diminish considerably from Fall to Winter and from 
Winter to Spring. 

(b) As the amount of time spent on the activities mentioned 

above in '»a" diminished over the school year, the follow- 
ing behaviors were observed more frequently in the USMES 
classes: writes/illustrates; reads-task;- f rc - reading, 
writing, drawing; small group-task (for development classes 
only); takes part in class discussion/presentation; listen/ 
look at class; listen/look at teacher, film, or lecture; 
and res ting/waiting/ fooling " around. 

•(c) Children in USMES Development classes spent significant 
amounts of time during the Winter observation period in 
the processes of constructing and working in small groups. 
Accordingly, they spent proportionately less time listen- 
ing to and looking at the teacher. (See lines 3, 16, and 
26 of Table 5.) 

(d) The amount of time students spent using How-To-Cards was 

negligible. This result from line 10 of Table 5 corrobor- 
ates the result of our teacher interviews on this point. 
Some interesting patterns of activities are noteworthy for the control 
classes. The results in Table 5 seem to indicate that: - 

(a) The children in USMES classes spent more time measuring, 
constructing, assembling, testing/experimenting, and re- 
cording data. This difference between USMES and control 
classes in time spent on physical activity was especially 
pronounced in the Fall observation period. 
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USMES students engaged in calculation to an appreciable 
degree during USMES classtime only in the Fall, The 
control students had a sustained emphasis on calculation, 
probably basic skills work in arithmetic. 
USMES Implementation classes spent virtually no time in 
any observation period on pre-structured writing during 
USMES class time, and the USMES Development classes de- 
voted only 2.1% of their time during the Winter observa- 
tions to pre-structured writing. However, control class 
children over the school year consistently spent about 
67o of their observed class time on pre-structured writing 
in workbooks or on worksheets. These results appear in . 
line 9 of fable 5. 

Line 15 of Table 5 suggests that during Winter and Spring 
observations about twice as much time as USMES children 
in one-to-one verbal interaction with their peers on social 
issues. 

As it was pointed out earlier in this report, the results 
on line 18 suggest that control teachers were using consider- 
able amounts of Class time in the Spring for review for con- 
trol students were then giving appreciably more pre-struc- 
tui-ed information to the teacher. The Spring column of line 
19 suggests that USMES implementation teacher were also 
spending more time in review and/or summary, but here the 
students are giving original information to their teachers. 
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Line 22 of Table 5 indicates that during the Fall observa- 
tion period, control children spent considerably more 
time taking part in class discussions or presentations 
than children in USMES Implementation classes. This 
pattern was reversed during the Winter when USMES Im- 
plementation class students spent 17% of the observed 
time participating in class discussion or making presen- 
tations to the class. Differences between USMES Imple- 
mentation and control classes in this behavior category 
were minimal in the Spring. 

Children in USMES Implementation classes spent appre- 
ciably more time listening to or looking at another child 
than control children did during the Fall and especially 
during the Winter observation periods. The group differ- 
ence recorded in Spring for this activity was very slight* 
These results appear in line 23 of Table 5. 
Not surprisingly, both USMES and control students spent a 
sizable percentage of time listening to and/or looking at 
their teachers. (See line 26 of Table 5.) More signifi- 
cant, however, is the result that control students spent 
fully 31% of their observed time in the Fall listening to/ 
looking at the teacher, while USMES Implementation students 
spent a much smaller percentage of their time (14%) in this 
way. The difference dropped from a ratio of 2:1 in the 
Fall (31% control versus 14% USMES), to a ratio of 3:2 in 
the Winter (30% versus 21%), to a 1:1 ratio in the Spring, 
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with no appreciable difference between USMES and control 
classes (287o versus 26%). These results suggest that, 
in the beginning, USMES teachers did in fact adopt less 
dominating roles. However, in the final stages of the 
units, the USMES teachers dominated class time to a much 
greater extent than they did earlier in the school year. 
(The USMES entry on line 19 for the Spring observation 
period showed a corresponding increase over the previous 
period.) 

(i) A final observation suggested by the data on line 28 of 
Table 5 is that both USMES and control students spent 
increasingly more time unproductively "resting/waiting/ 
fooling,** as the school year progressed. 

Chapter Summary 

USMES teachers varied considerably in how they used the USMES program. 
The diversity in application of USMES seemed especially noteworthy with re-- 
spect to two kinds of factors: (1) intensity of usage, and (2) teacher 
directiveness. Some USMES classes experienced brief applications of the 
program throughout the school year which others had their USMES time con- 
centrated in intensive periods over a few months only. Many combinations 
of levels of intensity and duration of usage were reported by the USMES 
sample teachers, and indeed the total number of hours they reportedly spent 
on USMES during the year rangjo from 8 to 108. 

Teachers* responses to the Program Monitoring Form and some of their 
comments during on-site interviews suggested that the factor of teacher 
directiveness could account for much of the additional observed and reported 
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variations in their application of the USMES program. Variability in 
teacher directiveness was evident (1) in the variety of ways teachers 
reportedly introduced the USMES challenge to their students, (2) in the 
statements of goals which the teachers established for their units, (3) 
in teachers* comments as to why and when they had to redirect their stu- 
dents* attention toward the primary unit challenge, and (4) in the tea- 
chers* assessments of the impact USMES usage has had on their teaching 
overall* 

Treatments and dosages could not be manipulated or controlled by the 
evaluators. The applications of USMES were diverse, and one can only 
assume that the control classes also represented much variety in their 
teaching/ learning experiences. However, our results from the Classroom 
Activity Analysis reported above indicated that there were clearly dis- 
tinguishable differences in the general patterns of activities which chac- 
terize USMES and those which characterize the control classes over the 
school year; During the Fall especially, USMES children spent a much 
larger portion of their time engaging in physi cal "hands on" activities, 
in testing and experimenting, and in collecting data. As the amount of 
time USMES classes spent on these activities diminished over the school 
year, increased amounts of time vrere devoted to the following behaviors: 
writing/illustrating; reading- task ; free reading, writing, drawing; taking 
part in class discussion/presentation; looking/ listening to other children 
and to the teacher. Control classes showed greater consistency in *:he 
patterns of student behaviors over the school year. Control students had 
a sustained emphasis on more structured, teacher-dominated activities? 
calculating; pre-structured writing in workbooks or on worksheets; and 
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listening to/looking at the teacher. 

Charters and Jones' (1973) caveat ''On the Risk of Appraising Non- 
Events in Program Evaluation" was cited in^the introduction to this re 
port. The foregoing discussion on the characteristics of USMES and 
control classes constitutes the USMES Evaluation Team's attempt to pro 
ceed to level 4, the most rigorous level of comprehensive program de- 
scription urged by Charters and Jones: 

"The manifest purpose of the teacher's role 
performance is to produce learning in students, 
but this cannot happen directly. The best 
the teacher can do is to induce statements to 
enpage in activities deemed instrumental to 
the covert psychological processes he hopes 
to affect. It is the student's own activities 
and experiences that are most immediately re- 
lated to learning ou tcomes , - • . . and it is of no 
small importance for program evaluators to 
attempt to describe or measure the school's 
educational program as experienced and enacted 
by students." (Charters & Jones, 1972, pp. 6-7.) 
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CHAPTER IV 

THE EFFECTS OF USMES ON STUDENT PERFORMANCE J.N PROBLEM 
SOLVING AND BASIC SKILLS 



Two kinds of data were drawn upon in the development of this chapter 
on the effects which the USMES program had upon its students. First, 
teacher perceptions of the program's effects on students were documented 
from interviews conducted during site visitations by the evaluation team. 
Second, student performance data in the areas of basic skills and problem 
solving were analyzed and discussed. Seriou;; limitations to the student 
performance data were noted, along with recommendations for resolution of 
these problems in future evaluation work. 

Tea cher Perceptions of Student Performance 

In order to assess the program from as many perspectives as possible, 
we felt that teachers would be questioned not only on their attitudes toward 
the USMES materials, the training workshops, etc., but also on their Per- 
ceptions of what was happening to student behavior as a result of the pro- 
gram. 

We inquired about those skills which USMES fostered or ignored, and 
yet we tried to avoid leading questions about specific kinds of skills; 
we encouraged the teachers to talk freely about those behavioral asp^cts 
of the child which were being developed or ignored under r.he influence of 
the program. We were impressed by the consistancy of their responses, re- 
gardless of the interviewer. 

The list of skills fostered by the classroom use of USMES, according 
to the teachers' perceptions, is quite long and can best be related to ^^e 
reader by grouping them into the following categories: 

(a) Most basic math/measurement skills: geometry,- fractions. 
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...^additions, subtractions, graphing. 
This appears to contradict another statement of the teachers.-that USMES 
does not replace math, and that basic skills must be taught first and then 
applied to USMES projects. Nevertheless, the above statement was made 
repeatedly. No teacher we interviewed felt C - '.nogram did not help 
foster these skills. Many felt they encourage! uhese behaviors even when 
not using the program, but admitted nonetheless that the program was still 
helpful in supporting these skills. 

(b) Work-study skills: evaluation, analyzing, synthesizing, 
use of the scientific method, ability to think, decision 
making, inquiry. 

(c) Socialization skills: working together in groups, foster- 
ing of cooperation rather than competition in trying to 
sulve problems. 

while the interviewers anticipated such responses as listed under category . 
#1, these tended not to be the skills the teachers felt to be the most im- 
portant. Rather their first response usually tended to be: "it fosters 
problem solving in the broadest sense," "encourages group processes," 
"decision making," "a sense of cooperation." While these are direct quotes 
from some interviews, these same ideas weire given over and over. 

(d) The development of a sense of confidence in the child. He 
experiments without a fear of failure. He senses accomp- 
lishment from the success he achieves. USMES develops per- 
serverenc^? in the child. 

(e) Wider definition of learning. Several teachers indicated 
that children are learn irg that -ihe whole world can be a 



classroom.. . that leaioing is not confined to the four 
walls of the classroom. 
None of the teachers felt that any specific skills were being totally 
ignored since, they reasoned, any skill can be built into the units. How- 
ever, they did sense some weaknesses in the program, weaknesses which 
might well be discussed here; 

One limitation, in their perceptions, is the kind of student who can 
respond to USMES. Students must have a sufficient degree of self -direction 
or ability to assume some responsibility to cope with this program. Mature, 
self-directed pupils do well with USMES; the converse is also true. Some 
students not only do not benefit, but would be better off using another pro- 
gram. Such students tend to look at any unstructured program as free time. 
For example, they cannot handle themselves in a Design Lab. They are unable 
to think of a problem worthy of solution, to say nothing of being able to 
get enough direction to solve it. Another aspect of this same set of per- 
ceptions of the teachers is the feeling that this program is inappropriate 
for inner-city students. The program is not relevant to their needs or 
interests. Location is probably not the key factor, however. This same 
criticism was voiced by a ceacher in an impoverished setting. 

Seeking a response as to whether or not USMES fosters a sense of re- 
sponsibility in students, we asked the question whether students seem more 
responsible for their learning as a result of using USMES. A large number 
of teachers felt that with USMES students must be responsible for their own 
learning. Students formulate their ovm problem and decide for themselves 
how it is to be solved. The teacher functions as a helper, not as a director. 
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Another large segment of the teachers agreed that children seemed re- 
sponsible for their learning, but were unwilling to say that it was the 
USMES program which contributed to his effect. In fact, some felt this 
would have happened in their classroom, even without USMES. Still another 
group felt the program did not enhance responsibility, .^s their students 
were not ready to accept responsibility for their learningo If these par- 
ticular students had had the opportunity to be in USMES several years in 
succession, behavioral changes might have occurred. Within the space of 
a year, however, noticeable changes did not occur. 

Answers to the question as to whether or not teachers encountered any 
different kinds of discipline problems because of USMES followed directly 
from the previous question on responsibility. The majority of teachers 
said they observed no difference between the kinds of discipline problems 
th' - had during USMES and during other classroom instruction. Soi'ae.even 
cited a lower rate of discipline problems during USMES because the students 
were more actively involved, or ^*too busy to cause any disturbances. 
Where children would not assume responsibility for their learning, how- 
ever, frequent discipline problems did arise during USMES. To these chil- 
dren, unplanned time was "free time," and these students cannot handle 
"free time" constructively. 

This brings us to the important questions: What effect has USMES had 
on children's behavior? ... on teachers behavior? While we have alluded to 
some of these earlier, they are included here as well since it is important 
to know what teachers feel are worth while changes in children and themselves 
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(1) Children enjoy learning more. They* re excited about 
school and working on their projects. They are not 
bored, but rather are enthusiastic about everything they 
do! 

(2) Communication between children, between children and 
teachers, between children and parents has been aided. 
By explanation, the teachers commented that children 
are excited about what they are doing and talk about 
it. Some teachers and administrators volunteered that 
parents also have commented on the children's enthusiasm 

. . over the program. 

(3) The children's self-confidence is increased; they know 
they can accomplich something. And because of this in- 
creased self-confidence, they try new things. 

(4) The students learn to use people other then the teacher 
as resources. They recognize that they can also learn 
from each other. 

With all these positive comments, the reader should be reminded that 
these teacher judgements apply to some children only. For others, the 
program seems to have few if any beneficial aspects. 

Measurement of Students' Problem Solving Abilities 

Tho primary objective of the USMES project is the enhancement of students* 
problem solving abilitie.^. The USMES approach requires that students them^ 
selves analyze a problem meaningful to them, identify variables or factors 
relevant to the solution of the problem, collect pertinent data, use the 
information to judge the correctness of the hypotheses, and for appropriate 
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conclusions and recommendations. 
A. Ins trumen ta t i on 

The Playground Problem was conceptualized according to the above 
description of the problem solving process. This test requires that 
students develop a plan for a playground which would serve the students 
in their school. A catalogue of equipment, cost data, and measuring 
instruments are given to the students along with the information that 
they could spend up to $2,000. The test designed for administi ation 
to five children randomly selected from a class, who work as a group 
toward the solution of the challenge. The Manual for the administration 
of the playground Problem is shown in Appendix E. 
B. Scoring and Scorer Reliability 

The scoring protocol developed for the Playground Problem yielded 
group scores on several dir.-.nsions. The behavioral assessment included 
rating scales of four aspects: motivation to accept the problem; conmit- 
ment to task; allocation of responsibilities for efficiency of manpower; 
and the nature of group leadership. The cognitive assessment included 
four summary rating scores on variable identification, measurement calcu- 
lation and recording. The students drawings of their proposed playgrounds 
were analyzed to yield four product scores: scale, labels, landmarks, and 
area designation. In summary then, 12 scores were derived from the scoring 
protocol: Eour behavioral, four cognitive, and four product scores. ■ The 
actual derivation of scores is described in the Scoring Manual shown in 
Appendix F. 

Two statff members scored the playground problems. Each worked inde- 
pendently to sccre the same ten, randomly selected, sets of text products. 
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(Each set included an audio tape of students' answers to questions, stu- 
dents' note papers, and the students^ playground drawing.) Having com- 
pleted this scoring, the two staff members compared ratings and discussed 
any discrepancies. Th6y established points of agreement on how they would 
score various situiitions or various responses. Then the scorers proceeded 
to complete the remainder of the scoring without duplicating efforts and 
with consultation only on difficult judgements. 
C. Results 

Sample classes were selected to include a cross section of grade levels, 
USMES units, socioeconomic levels, and geographical areas within each of 
the following treatment designations: USMES Development classes (25); USMES 
Implementation classes (18); and control classes (18) at the same grade 
level in the same school as the Implementation classes. The Development 
classes had no controls. 

Completepre- and post-test returns were obtained for only 38 classes. 
The distribution of these classes by treatment and by grade level is shown 
in Table 6. 

1, Behavioral Aspects , Distributions of pre- and post-test rating for 
Development, Implementation and Control Classes on each of the four behavioral 
aspects are shown in Table 7 through 10. Chi squares were computed to deter- 
mine if there were significant differences in ratings for the pre-test or 
the post- test, among the treatment groups. None was significant. 

Table 7 which contains the pre-- and post-test ratings on the aspect of 
motivation to accept the problem shows a slight increase in the number of 
implementation class students who initially attempted to solve the^ problem. 
The motivation ratings for development and control groups remain virtually 
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TABLE 6 



Distribution by Treatment and Grade Level 
. Sample Classes with Pre- and Post-Test 
Results on the playground Problem 



Treatment 


Grade Levels 


Total 




Primary 
1,2,3 


Intermediate 
4,5,6 


Advanced 
7,8 




Development 


4 


13 


1 


18 


Implementation 


1 


7 


2 


10 


Control 


1 


7 


2 


10 


Total 


6 


27 


5 


38 



93 



TABLE 7 



Distributions of Pre^Test and Post- Test Ratings on Motivation to 
Accept the Problem for Development, Implementation 
and Control Classes 



Treatment Group 



Development 
(N=18) 

Implementation 
(N=10) 

Control 
(N=10) 




Ratings 





11% 


6% 


83% 




11% 


6% 


83% 


107o 




40% 


50% 






1C% 


90% 




10% 




90% 


107o 






90% 
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TA1^.LE 8 



Distributions o£ Pre-Test and Post-Test Ratings 
Connnitment to Task for Development, 
Implementation, and Control Classes 



■• • cjatment Group 


Test 


Ratings 






0 


1 


2 


3 


4 


5 


Development 
(N=18) 


Pre 
Post 




6% 


17 /o 
67o 


507. 
337. 


117= 
287. 


227. 
287. 


Implementation 
(N=10) 


Pre 
Post 






207. 


607. 
507. 


107. 
207. 


107. 
307c 


Control 
(N=10) 


Pre 
Post 






207o 


607. 
507. 


407. 
107. 


207. 
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TABLE 9 

Distributions, of Pre-Test and Post-Test Ratings 
on Efficient Allocation of Responsibilities 
for Development, Implementation, and 
Control Glasses 





Treatment Group 


Test 


Ratings 






0 1 


2 


3 


4 


5 


Development 
(N=»18) 


Pre 
Post 


11% 
11% 


17% 
17% 


50% 
17% 


11% 
28% 


11% 
28% 


Implementation 
(N=10) 


Pre 
Post 


10% 2,0% 


^0% 
30% 


30% 


20% 


10% 
40% 


Control 
(N=10 


Pre 
Post 


20% 


40% 
40% 


50% 
20% 


10% 
10% 


10% 
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TABLE 10 



Distributions of Pre-Test and Post-Test Ratings 
oh Group Leadership for Development, 
Implementation, and Control Classes 



Treatment Group 


Test 


Ratings 






0 


1 


2 


3 


4 


Development 
(N=18) 


Pre 
Post 


67, 
67o 


67o 


307= 
337= 


227o 
22% 


397o 
337o 


Implementation 
(N=10) 


Pre 
Post 




107o 


107= 
307= 


207o 
307o 


607o 
40% 


Control 
(N=10) 


Pre 
Post 


107= 
107= 


107o 


307= 
307= 


307= 
207= 


20% 
40% 
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- The second behavioral aspect rated was commitment to task, or the 
intensity and sustenance of group interest in arriving at a solution. 
Table 8 shows that the implementation groups increase slightly on this 
factor from pre to post administration. The distribution of ratings for 
development classes showed more variability on the post-test with a very 
slight increase toward the high end of the scale. 

Table 9 shows the distributions of group ratings on the behavioral 
aspect of allocation of responsibilities for efficiency of manpower. The 
Implementation classes showed a decrease in efficiency from pre-test to 
post-test. Overall, the classes in all three treatment groups became more 
variable on the post-test. 

The fourth behavioral aspect scored was the nature of group leadership. 
Table 10 shows that the overall patterns across classes within each treatment 
designation changed very little from pre- to post-test. In most groups 
some form of leadership behavior emerged which was other than autocratic. 

2. cognitive Aspects . The scoring protocol for cognitive aspects of 
the students problem solving behaviors involved coding the variables or 
factors which each group identified as salient to the solution of the Play- 
ground Problem. Up to ten variables were scored. Eight possible factors 
were anticipated for the scoring protocol. Two additional variables-could 
be accommodated. The number of factors which each group identified for 
~conside^?^on were sumn^ed. These scores'Tre tem^^nH^T^f-rd'^^rtrf i^^¥ti3S" ' 
scores. No group identified more than 10 factors. 

Summations for each group were made for the levels of measurement the 
group achieved for each variable they identified. Similarly, summations 
were obtained across calculation ratings for each variable and across ratings 
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on the adequacy of data recordings. Data analysis was based on these four 
summary measures: identification, measurement, calculation and recording. 

Two types of analyses were conducted* First repeated measures analyses 
of variance were conducted to determine if Development, Implementation and/ 
or control classes realized statistically significant gains in any of the 
four cognitive summary measures. .Second, covariance analyses were used to 
test the hypothesis that thiire were no statistically signific differences 
in post-test dilierence among the groups* 

Sources of variance tables for the four repeated measures analyses using 
each of the four suTranary measures as dependent variables are shown in 
Tables 11, 12, 13 and 14. No significant changes from pre- to post-test were 
noted for the measurement, calcul a tion , and recording scores. This result- 
applied to all three treatment groups. 

Significant differences in identif icatioii scores from pre- to post-test 
administration^were observed (p <.001). An examination of Table 11 further 
reveals that this change from pre- to post-test scores must be qualified by 
treatment group. In fact, all three treatment groups showed a decline in 
the average number of factors they identified for consideration in their 
solution to the Playground Problem. Inspection of the means in Table 15 
reveals that while the decline for the Development classei^ was very slight, 
the decline in the number of variables which the Implementation and control 
classes identified was more pronounced. These latter two groups identifi'id 
on the average, approximately five factors for consideration on the pre-test 
but only about 3.5, on the average, for the post-test. The means summarized 
Table 15 are also portrayed graphically in Figures 1-4. 
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TABLE 11 



Repeated Measures Analysis of Variance for Pre- and Post-T 
Results of the Identification Scores for Development, 
Iraplementation, and Control Classes 



Sources of 
Variance 


'df 


Sum of 
Squares 


Mean 

gqUarcb 


r wi LIU 


p 


Between Subjects 


31 


121.632 


3.287 






Treatment (T) 


2 


0.131 


0.066 


0.0189 


.982, 


Subjects X T 


35 


121.500 


3.471 






Within Subjects 


38 


66.000 


1.737 






Test Administration (A) 


1 


15.210 


15.210 


' 12.8280 


.001 


Tx A 


2 


9.290 


4.645 


3.9170 


0.028 


Subjects X A X T 


35 


41.500 


1.186 






Total 


75 


187.632 


2.502 
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TABLE 12 



Repeated Measures Analysis of Variance for Pre- and Post- 
Results of the Heasurement Scores for Development, 
' Implementation, and Control Classes 



ERIC 



Sources of Sum of Mean 

Variance df Squares Squares F-Ratio 



Between Subjects 


37 


10006.738 


27.209 






Treatment (T) 


2 


13.180 


6.590 


0.232 


.796 


Subjects X T. 


35 


993.559 


28.387 






Within Subjects 


38 


218.000 


5.737 






Tests (A) 


I 


6.367 


6.367 


1.149 


.291 


T X A 


2 


17.828 


8.914 


1.610 


,213 


Subjects X A x T 


35 


193.805 


5.537 






Total 


' 75 


1224,738 


16.330 ■ 
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TABLE 13 

ited Measures Analysis of Variance for Pre- and' Pos 
Results of the Calculation Scores for Development, 
Implementation, and Control Classes 



Sources of Sum of ' Mean 

Variance df Squares Squares P-Ratio 



Between Subjects 37 694.855 18.780 

TreaM(T), ' 2 4.605 2.303 0.116 0,890 

Subjects xT 35 690.250 19.721 

Within Subjects 38 247.500 6.513 

Tests (A) 1 1.066 1.066 0.155 0.698 

2 5.628 ,2.814 0.409 0.673 

Subjects X Ax T 35 240.806 6.880 

Total 75 942.355 12.565 



TABLE 14 



.. Repeated Measures Analysis of Variance for Pre- and Post-Test 
Results of the Recording Scores for Development, 
Implementation, and Control Classes 



Sources of 




Sum of 


Mean 


F-Ratio 


0 

r 


Variance 


df 


Squares 


Squares 


Between Subjects 


37 


424.737 


11.479 






Treatment (T) ' 


2 


14.948 


7.474 


0.638 


■ .539 


Subjects X T 


35 


409.789 


11.708 






.Within Subjects 


38 


107.000 


2,816 






Tests (A) 


I 


2.579 


2.579 


0.890 


.646 


■ TxA 


2 


3.032 


1.516 


0.523 


,603 


Subjects X A X T 




- 101.389 


2.897 






Total 


75 


531.737 


, 7.090 







TABLE 15 



Summary of Means and Standard Deviations for. each Treatment 
Group on Each of the Playground Problem 
Cognitive Variables 



Treatment -Gifoup 



Developmental Implementation Control 







X 


sd 


X 


sd . 


X 


sd 


Identification 
(Maximum 

possible 

score=10) 


Pre 
Post 

Adjusted 


4.33 
4.17 
4.42 


1.64 
1.38 


5.00 
3.60 
3.52 


1.41 
1.35 


5.20 
3.50 
3.33 


1.40 
1.90 


Measurement 
(Maximum 
possible 


Pre 
Post 

Adjusted 


9.44 
9.78 
9.86 


4.79 
3.86 


9.70 
8.90 

8.83 


3.09 
3.25 


9.60 
7.60 
7.59 


4.43 
4.55 


score=50) 
















Calculates 
(Maximum 
possible 


Pre 
Post 

Adjusted 


6.06 
16.28 
6.13 


3.52 
3.06 


5.20 
6.20 
6.42 


2.35 
3.49 


5.90 
5.40 
5.32 


5.36 
3.98 


score=27) 
















Records 
(Maximum 
possible 


Pre 
Post 

Adjusted 


4.00 
3.39 
3.00 


3.20 
2.35 


2.80 
3.10 
3.35 


1.75 
2.13 


3.00 
2.40 
2.54 


3.16 
3.06 


score=16) 

















108 



-89- 



6 



5 



4 .. 




lental 



Implementation 
Contro 1 
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Figure 1. --Graph of Pre-test and Post- test Means of 
Identification Scores for Development, 
Implementation, and Control Groups. 
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11 



10 




Developmental 



Implementation 



Control 



Pre 



Post 



Figure 2. --Graph of Pre-test and Post-test Means of 
Measurement Scores for Development, Im-- 
plementation, and Control Groups 
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Post 



Pre 



Figure. 3. --Graph of Pre-test and Post-test Means of 
Calculation Scores for Developrment , Im- 
plementation,, and. Control Groups. 
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Developmental 
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C ' ntr.o 1 
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Pre 



Post 



rigure 4.--Graph of Pre-test and Post-test Means of 
Recording Scores for Development, Im- 
plementation, and Control Groups. 
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The evaluation team had hypothesized that this drop in identification 
scores from pre- to post-test would be observed for USMES Implementation 
classes in which most of the students had no prior USMES experience. How- 
ever, we hypothesized too a corresponding increase in their measurement 
and calculation scores, the rationale being that USMES students might pur- 
sue fewer issues in greater detail in a timed test situation. The results 
do not support this hypothesis. 

Results of the one-way analyses of covariance used to test for adjusted 
post-test differences among the three treatment groups are shown in Table 16 
for identification, measuremene", calculation and recording scores. No sig- 
nificant difference -^'^^^ L>bi;erved amsmg treatment groups for any of the 
four dependent variabM^:>^ So conclusions could be drawn about the relative 
superiority of the tt^atsssrit groups with respect to the cognitive factors 
involved in problem solvmg as measured by the Playground Problem and its 

"t 

scoring protocol. 

3. Product Asp€ ^_s. Mstribution of pre- and post-test ratings for 
Development, Implem. Jcation, and control classes on each of the four product 
aspects are shown it, Xatles 17-20. Chi squares were computed to determine 
if there were signif . r ant differences among treatmenc; groups on any of feijie 
pre-test or post-te .t praa-ct ratings. None of the c -i square results imn 
s ignif ic." (\t. 

Measu r ement of Students' }>a>iic Skill Development 

As was the case i» the 1972-73 evaluation of the USMES program, per- 
formance in basic skil • R ireas oE children exposed to USMES was an integral 
component of the 1973-74 evaluation. In the 1972-73 evaluation the results 
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TABLE 16 



Analysis of Covariance for Development, Implementation, 
and Control Classes Using the Four Post-Test Cognitive 
Summary Ratings as Dependent Variables and the 
Corresponding Pre- Test Scores as Covariates 



Source of Variance 


df SS 


MS 


F 


P 


Measurement Scores 


Treatment 

Error 

Total 


2 8.90 
34 61.78 
36 70.68 


4.45 
1.82 
1.97 


2.45 


0.10 


Identification Scores 


Treatment 

Error 

Total 


2 33.57 
34 289.51 
36 323.08 


16.79 
8.52 
8.97 


1.97 


0.16 


Calculation Scores 


Treatment 

Error 

Total 


2 6.70 
34 314.40 
36 321. LO 


3.35 
9.25 
8.92 


0.36 


0.70 


Recording Scores 


Treatment 

Error 

Total 


2 3.28 
34 138.04 
36 141.32 


1.64 
4.06 
3.93 


0.40 


0.67 
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TABLE 17 



Distributions of Pre- Test and Post-Test Ratings 
Product Scale for Development, Implementation, 
aud Control Classes 



Treatment Group 


Test 


Ratings 






0 


1 


2 


DeveLapment 
(N=18): 


Pre 
Post 


507, 
227, 


447. 
117, 


67, 
177, 


Implementation 
(N=LO) 


Pre 
Post 


107, 
207, 


807, 
707, 


107, 
107, 


Control ■ 
(N=l:0) 


Pre 
Post 


407, 
30% 


607, 
707, 
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TABLE 18 



Distributions of Pre-Test and Post-Test Ratings on 
Product Labels for Development, Implementation, 
and Control Classes 



TEeatment Group 


Tiest 


Ratings 






0 


1 


Development 
(N=18) 


Pre 
Post 


67% 
30% 


33% 
70% 


Imp Ijemen t a t i on 
(N=10) 


Pre 
Post 


30% 
30% 


70% 
70% 


Con tro 1 
(N=lO0 


Pre 
Post 


20% 
50% 


80% 
50% 
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TABLE 19 



Distributions of Pre-Test and Post-Test Ratings on Prod- 
Landmarks for Development, Implementation 
and Control Clasoes 



Treatment Group 


Test 


Racings 






0 


1 


2 


Development 
(N=18) 


Pre 
Post 


61% 
67% 


L7% 
22% 


22% 
11% 


Implementation 
(N=10) 


Pre 
Post 


80% 
70% 


20% 

20% 


10% 


Control 
(N=10) 


Pre 
Post 


80% 
70% 


10% 
30% 


10% 
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'L'ABLE 20 

Distributions of Pre-Test and Post-Test Ratings 
Product Area Designation for Development, 
Implementation, and Control Classes 



Treatment Group 


Test 


Ratings 






0 


1 


Developmen t 
(N=18) 


Pre 
Post 


50 
56 


50 
44 


Implementation 
(N=10) 


Pre 
Post 


20 
50 


80 
50 


Control 
(N=10) 


Pre 
Post 


40 
40 


60 
60 
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indicated that there was no consistent evidence which could lead one to 
conclude that using USMES hindered growth in the elementary school basic 
skills- areas (i.e., reading and math) . The purpose of again examining 
growth in the basic skills areas was to determine if the 1972-73 results 
would br replicated. 
A. "Procedures 

To answer the questions concerning growth in the basic skills areas, 
a two group (i.e., USMES and non-USMES classes), pre-test/pos t-test design 
was :employed. The USMES classes received the treatment (i. e. , worked on 
anlTSMES challenge), while the non-USMES (Control) classes did not. Thus, 
the independent variable was exposure or non-exposure to USMES. The de- 
pendent variable was the score each child attained on Form A of the ,1973 
version of the Stanford Achievement Test. The measurement of dependent 
variables (i.e., administration of the SAT subtests) in the sample classes 
was done both in the Fall and in the Spring. Unfortunately no standardized 
testing schedule was employed and thus both pre-tests and post-tests were 
giv«n over a 2-3 month period. This problem reduces the generalizabi li ty 
of results as one can not really talk of "gains over the school year," but 
only of "gains over the pre- test/post-test period." The administration of 
these tests was limited to USMES implementation classes in both the National 
and Chicago samples. Due to a multitude of problems encountered with the 
Chicago Sample, only the results of the National Sample will be presented. 
The ISatlonal Sample consisted of 18 experimental (i.e., USMES) and 18 con- 
trol (i^e. , non-USflES counterpart) classes. Due to several data collection 
pxdblems:, mentioned earlier, data from only six USMES classes and their con- 
troILs. could be used in the analysis, 
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Of the entire Stanford Achievement Test Batcery, only the Reading 
Comprehension and Mathematics Computation subtest were administered... 
These were the basic skill areas which were of primary concern to prospec- 
tive USMES users, and schools were reluctant to submit to extensive stan- 
dardized testing. 

These tests were administered by either the classroom teachers or. the 
USMES classroom observers. These observers were employed by uhe USMES 
program for this purpose as well as for other data collection. The choice 
of who administered the tests was left to the discretion of the teachers 
and observers. 

In the second grades, the Primary II form of the SAT's was used, and 
in the third grade the Primary III form was used. For both of these forms, 
the students marked their responses in the test booklets. In the fourth 
grade the Intermediate I form was used while the Intermediate II form was 
selected for fifth and sixth grade classes, and the Advanced form for the 
seventh and eighth grade classes. For the Intermediate I, Intermediate II, 
and Advanced forms, the students responded directly on machine scorable 
answer sheets. The data from the Primary II forms was transferred from 
the test booklets to Digitek answer sheets for optical scanning. All data 
from the remaining three forms was key punched directly from the answer 
sheets themselves. Test scoring was done with a computer program prepared ; 
especially by the program evaluators. The end results of^ the scoring pro- 
cedure were, for each student, a raw score and it's corresponding scaled 
scqre on each of the two subtests of interest. The calculation of the scaled 
scores is described in the following paragraph which is taken from the Norms 
Booklet , Form A, of the Stanford Achievement Test. 
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Scaled scores on the Stanford Achievement test 
were obtained through a computerized application 
of the Thurstones absolute scaling procedure. 
This resulted in the development of a system of 
inter-battery standard scores which permitted 
the translation of raw scores at each level to 
standard scores with comparability across levels 
for a test area. The scale values were derived 
by setting the median raw scores of grade 3 and 
grade 8 in the Fall standardization equal to 132 
and 182 respectively. 

The scaled scores, rather than the raw scores were used ii; all anal- 
yses. The analyses were done using packaged statistical programs avail- 
able at the Boston University Computer Center. 

As had been expected, there was, in each classroom, some loss of data 
from pre-test to post-test due to both absenteeism and the mobility of the 
student sample. In addition, due to the small number of analyzable class- 
rooms, students rather than the classrooms were used as the unit of analysis, 

r 

although the classroom were originally designated the sampling unit. 

One other point which needs to be mentioned concerns the classes from 
the National Sample which were analyzable. Four of the six classes are 
from the same geographical area, a fact which makes generalizations to the 
USMES program in other areas of the country impossible. 
B. Results 

The data were submitted to two general analyses: (I) repeated measures . 
analysis of variance, and (2) analysis of cbvariance. The repeated measures 
analysis of variance was used to answer the question, »»Do the USMES and/or 
Control (i.e., non-USMES) classes realize statistically significant gains 
in mathematics (i.e., Reading Comprehension) from pre- test to post- test." 
The analysis of covariance was used to test the hypothesis that there were 
no statistically significant differences in post-test scores between the 
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USMES and Control classes once adequate statistical allowances were made 
relative to pre-test differences between the groups. The pre-test score 
was used as the covariate. The results will be presented below in two 
sections corresponding to the two Stanford Achievement Tests subtests ad- 
minisf.ered: i. ^., Reading Comprehension and Mathematics Computation. 

1. Re ading Comprehension . The Reading Comprehension ; subtest measures 
reading comprehension at levels varying from simple recognition to making 
inferences from several related sentences in varying content areas. 

The test questions sample the following skills: 

- Comprehension of global meaning. 

- Comprehension of the meaning of detailed information. 
-Comprehension of implied meaning. 

-Use of context for word and paragraph meanings. 
-Drawing inferences from what has been said. 
The difficulty of the items and the length of the selected paragraphs in- 
creases from the Primary II through the Advanced forms. The time allowed for 
the different levels are as follows: Primary II, 45 minutes; Intermediate I, 
35 minutes; Intermediate XI, 35 minutes; and Advanced 35 minutes. 

a. Within group diiif f erences . Pre-test/Post-test gains were ana- 
lyzed for both USMES and Control samples and for each individual classroom. 
Means and standard deviations for each classroom are presented in Table 21. 

The classes were grouped into three general categories of grade levels 
based on whether they were administered Primary, Intermediate, or Advanced 
forms of the SAT's. The pair of classes lettered "A" is a Primary Level 
class, pairs B, C, and D are Intermediate Level classes, and pairs E and 
F are Advanced Level classes. An examination of the pre-test and post-test 
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TADLE 21 

Means and Standard Deviations for 
Pre- Test and Post-Test Reading 
Comprehension 



Pairs 


Place 
Li J. d a a 


N 


Pre-Test 


Post-Te£.t 


Adjusted 


P r imary 


USMES^ 


18 


122.72 
(14.06)^ 


136.44 
(19.17) 


132.2^ 


A 


Control^ 


23 


114.35 
(28.75) 


128.22 
(31.39) 


132.4 


T n ^ fsTTTipdi a tf» 

X il U C i< llIC VI X ct I. c 

B 


USMES^ 


24 


141.29 
(16.17) 


158.67 
(10.66) 


160.0 


Control 


18 


145.11 
(24.42) 


154.06 
(28.00) 


152.7 




USMES^ 


19 


167.74 
(16.43) 


176.68 
(22.77) 


179.1 


C 


Control^ 


24 • 


172.08 
(21.21) 


179.00 
(27.96) 


176.6 




USMES^ 


25 


116.04 
(43.15) 


148.72 
(20.45) 


152.0 


D 


Control 


26 


149.23 
(17.75) 


152.31 
(17.11) 


149.0 


Advanced 


USMES^ 


19 


191.89 
(33.66) 


206.00 
(18.85) 


200.6 


E 


Control^ 


32 


168.97 
(21.67) 


179.25 
(17.80) 


184.7 




USMES^ 


22 


201.68 
(18.08) 


.215.45 
( 6.96) 


215.1 


F 


Control 


27 


199.85 
(20.04) 


209.41 
(12.90) 


209.8 



a. Figures in parentheses are standard deviations 

b. Represents post-test means adjusted for pre-test differences 

c. Significant at or beyond the .05 level 
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means for each class indicate that all classes realized gains in reading, 
comprehension, and for all classes except the control class of pair D, 
these gains were statistically significant at or beyond the ,05 probabil- 
ity level. Repeated measures analysis of variance results for each class 
can be found in Tables 22 through 27. Overall the pre-test/pos t-test gains 
in mean scores varied from approximately 9 points (USMES class C/Intermediate 
Level) to 32 points (USMES class D/Intermediate Level) for tho USMES classes 
while the corresponding mean increases for the control classes was approxi- 
mately 3 points (Control class D/Intermediate) to 14 points (Control class 
A/Primary) . 

Tables 28 and 29 present the repeated measured analysis of variance of 
the total USMES group (Table 28) and the total Control group (Table 29). 
Both groups realized statistically significant gains in reading comprehen- 
sion over the pre- test/post- ast assessment period. 

In summary, the USMES classes realized significant gains in Reading Com- 
prehension over the period during which they were studied. The exposure to 
the USMES challenge does not appear, in anyway, to have hindered growth in 
this basic skills area. This appears to be true for virtually all grade 
levels analyzed. Control classes also appear to realize gains. 

b. Between group differences . A two-factor analysis of covariance 
was used to investigate differences in Reading Comprehension between the 
USMES and Control clas&es. The first factor was Treatment (USMES vs. non- 
USt'ES) and the second factor was Grade level (Primary, Intermediate, and 
Advanced). Table 30 presents the combined means for each factor. Table 
31 contains the summary table of the analysis of covariance. An examination 
of the F-Ratio and their associated probabilities reveals statistically 
significant main effects for both Treatment and Grade Level. The interaction 
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TABLE 22 



Repeated Measures Analysis of Variance Treatment (USMES, Control) 
by Assessment (Pre-Test, Post-Test) Reading 
Comprehension for Pair A 



Sources of 
Variance 



df 



Sum of 
Squares 



Mean 
Squares 



F- Ratio 



Between Subjects 


40 


4I771.00 


1194.275 






Treatment (T) 


1 ' 


1391.00 


1391,000 


1.170 


.286 


Subjects X T 


39 


■ 46380.00 


1189,231 






Within Subjects 


41 


6999.00 


170.707 






Assessment (A) 


1 


3906.00 


3906,000 


, 49.267 


.000 


T X A 


1 


1.00 


1,000 


■ 0.013 


.906 


Subjects X P X T 


39 


3092.00 


79.280 . 






Total 


81 


54710.00 


676.173 







I 

H 

0 

Ui 

I 



15 
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TABLE 23 



id Heasures Analysis c; variance Treatraeit |l8MES, 
by Assessment (Pis- ' 2St, Post-Test) Ws^n 
Comprehefeaon for Pair B 



in es 



df 



Sui of 
Squares 



Hean 
Squares 



F-Ratis 





41 


27524.00 


671.317 




atiitient (T) 


\ 


3.00 


3.000 


O.004 




40 


275ZL00 


688,025 






42 


8909.00 


212.119 




. vSei;Sii£nt (A) 


I 


3976.00 


3976.Q;00 


34.814 




1 


366.00 


366.000 


3.206 


Bl :)ects X P X T 


40 


4567.00 


114.175 




To;t3i 


83 


36433.00 


438.952 





TABLE 24 



Repeated .^. 'asures Analysis of Variance Treatment (USMES, Control) 
by Asssssmest (Pre-Test, Posj^^'Jst) Reading 
Comprehension for&i: C 
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Sources of Sum of 

Variance df Squares Squares F-Ratio 



Between Subjects 


42 


38205*00 


909.643 






Treatment (T) 


1 


235.00 


23:5.000 


0.254 


.622 


Subjects X T 


41 


379?0,00 


926.097 






Within Subjects 


43 


5881.00 


B6.,76] 






Assessment (A) 


1 


1.311.00 


Bii.,.000 






Ix A 


I 


24.00 


24000 


11.824 


.002 


Subjects X P X T 


41 


4546.00 


m.878 


0.217 


.649 


Total 


85 


44086.00 


:5m.;659 







0 
I 
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RepeatedSeasures Analysis of Variance Treatment (USMES, Control) 
by Assessment (Pr.fi.», Post-Test) Reading 
ComprehenJion for Pair D 



Sources oi 
Variance 


df 


Sun o f 
Squares 


iMean 
Squataes 


F-Ratio 


P 


Between SubjiecB- 


50 


54119.00 


:i082,.3B0 






Treatment (I) 


1 


8:619.00 


8619.10 


'W82 


.003 


Subjects X 1 


49 


45500.00 


928.:571 






Within SuhjecB^ 


■51 


37890.0B 


742.941 






AssesSHEnt (i) 


1 


7887,0s 


7887..000 


15.828 


.000 


T xi 


\ 


5586.01} 


5586.000 


11,210 


.;002 


Subjeits X P I T 


49 


2441].0'0 


498.306 






Total 


IGl 


920O9J0 


9ilS.980 







TABLE 26 



Repeated Heasures Analysis of Variance Treatment (USMES, Control) 
by Assessment (Pre-Test, Post-Test) Reading 
•Comprehension for Pair E 



Sources of 




Sum of 


Mean 






Variance 


or 




Sfliiares 


F-Ratio 


P 


Between Subjects 


50 


56858.00 


1137.160 






Treatment (T) 


1 


14711.00 


14711.000 


17.103 


.000 


Subjects X T 


49 ' 


42147.00 


860.143 






Within Subjects 


51 


12604,00 


247.137 






Assessment (A) 


1 


3496.00 


3496.000 


18.987 


.000 


TxA 


1 


86.00 


86.000 


0.467 


.505 


Subjects X P X T 


49 


9022.00 


184.122 






Total 


101 


69462.00 


687.742 







TABLE 21 

Repeated Measures Analysis of Variance Treatment (0SMES, 'Control) 
by Assessment (Pre- Test, Post-Test) Reading 
Comprehension for Pair F 



Sources of 




Sum of 


Mean 




P 


Variance 


df 


Squares 


Squares 


F-Ratio 


Between Subjects 


48 


19215.00 


400.313 






Treatment (T) 


1 


376.00 


376.000 


0.938 


.661 


Subjects X T 


47 


18839.00 


400.830 






Within Subjects 


49 


7139.00 


145.694 






Assessment (A) 


1 


3211.00 


3211.000 


39.497 


.000 


Tx A 


1 


107.00 


107.000 


1.316 


.256 


Subjects X P X T 


47 


3821.00 


81.298 






Total 


97 


26354.00 


271.691 







TABLE 28 



One-way Repeated Measures Analysis of Variance 
for Reading Comprehension, 
USMES 



Sources of 
yariance 

.Within Subjects 
; Assessment (A) 
; Error 

■Total 



df 

126 
1 

126 
253 



Sum of 
Squares 

325289.00 
19437.00 
38886.00 

383621.00 



Mean 
Squares 

2581.73 
19437.00 
308.62 



F-Ratio 



62.981 



.000 



31 



TABLE 29 



One-Way Repeated Measures Analysis of Variance 
for Reading Comprehension 
Control 




Sources of 
Variance 



df 



Sum, of 
Squares 



Mean 
Squares 



F-Ratio P 



Within Subjects 
Assessment (A) 
Error 



149 333862.00 2240.68 
I 5745.00 5745.00 
149 15377.00 103.20 



55.67 .000 



I 

H 
H 
W 
I 



Total 



299 354984.00 
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TABLE 30 



Combined Pre-Test, Post-Test, and Adjusted Means for 
Treatment (USMES and Control) and Grade Level 
(Primary, Intermediate, and Advanced) 
^ Reading Comprehension 



Group 


N 


Pre-Test 


Post-Test 


Adjusted 


USMES 


127 


155.7 


173.2 


168.6 


Control 


150 


160.4 


169.1 


161.7 


Primary 


41 


118.0 


131.8 


152.1 


Intermediate 


136 , 


147.8 


161.1 


163.7 


Advanced 


100 


188.9 


200.4 


179.7 



140 



TABLE 31 



Two-Factor Analysis of Covariance for Reading, Comprehension, 
Treatment (USMES vs. Control) by Grade Level 
(Primary, Intermediate, Advanced) 



Sources of 




Sum of 


Mean 




P 


Variance 


df 


Squares 


Squares 


F-Ratio 


Treatment (T) 


1 


3171.812 


3171.812 


12.218 


0.0006 


Grade (G) 


2 


14096.938 


7048.469 


27.151 


0.0001 


Tx G 


2 


296.562 


148.281 


0.571 


0.566 


Error 


270 


70092.540 


259.602 






Total 


275 


87657.852 


318.756 
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of these factors was not significant. These results indicate (1) that 
significantly higher scores in Reading Comprehension once pre-test differ- 
ences were taken into account, and (2) that there were statistically sig- 
nificant differences among the three grade levels, with the rank from 
highest adjusted mean to lowest being Advanced, Intermediate and Primary. 
These adjusted means on the SAT Reading Comprehension subtest are presented 
graphically in Figure 5. 

A summary of the reisialts of covariance analyses for the indlividual classes 
is presented in Table 32. The data in tils table imiiicate that only at the 
Advatrrsd L^el (pairs E and F:) were there statistical-ly siignlflcant differ- 
ences Reading Comprehension between USMES and Contzrol classes. Refer- 
ences t:o the Adjusted means column of Table 21 sheds further' light on these 
findings. In all classes, except class A (Primary Level), the adjusted means 
for USMES classes were higher than those for the Control classes. The dif- 
ference in adjusted means for pair B appears at first glance to be large 
enough to reach statistical significance, however, reference to Table 32 re- 
veals that for pair B, there was a relatively large amount of variance with- 
in the groups, thus increasing the size of the error term. Consequently, 
the ratio of the variance between the groups to the variance within the 
groups was not large enough to reach statistical significance at the con- 
ventional level (.05) , ^they were rather large and are worthy of note. 

These findings would seem to indicate that in the six classes analyzed, 
the exposure to the USMES program does not hinder growth in Reading Compre- 
hension, and in some cases aids growth, relative to Control classes which 
were not exposed to this program. 
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Figure 5. --Adjusted Cell Means by treatment and on 
the SAT Reading Comprehension subtest. 
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TABLE 32 



Oneway Analyses of Covariance -by Class 
for the Reading Comprehension 
Subtest. 



Pair 



Sources of 
Variance 



df 



Sum of 
Squares 



■Mean ' 
Squares 



F-Ratio 



Primary Level 
A 



Intermediate .Level 
B 



I 



Advanced Level 
E 



4 o 



fll^ Amah a M f* 

Treatment 


1 


\jtJy 


0.34 


0,002 


.964 


Error 


38 


6181,80 


162.68 






Total 


39 


6182.15 


158.52 






Treatment 


1 


547.08 


547,08 ■ 


2.742 




Error 


39 


7780.50 


199,50''' 






Total 


40 


8327.58 


218.19 






Treatment 


1 


63.55 


63.55 


0.285 


.597 


Error 


40 


8931.52 


223.29 






Total 


41 


8995.07 


219.39 






Treatment 


1 


94.92 


94.92 


0.299 


0.587 


Error 


48 


15241.10 


317.52 






Total 


49 


15336.02 


312.98 






Treatment 


1 


2549.91 


B49.91 


14.638 


.0004 


Error 


48 


8361.60 


04.20 






Total 


49 


109U.51 


.:222.68 






Treatment 


1 


333.63 


333.63 


7.365. 


,009 


Error 


■46 


2083,66 


45.30 






Total 


47 


2417,30 


' ■51.43, 







I 

H 
H 

1 
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2. Mathematics Computation . All forms of the Mathematics Computation 
Subtest of the SAT measures skills in the four basic operations - addition, , 
subtraction, multiplication, and division. The size of the numbers used ., 
in the problems become larger as one progresses from the Primary II test / / 
through the Advanced test. In addition to basic computation, there are 
questions requiring knowledge of greater-than and less-than relationships, . 
common and decimal fractions, percent, average, exponents, sin^.olification 
of sentences, and graphing. The amount of time allowed for each form are 
as follows: Primary II, 38 minutes; Intermediate I, 35 minutes; Interme- 
diate II, 35 minutes; and Advanced, 35 minutes. 

The USMES class of Pair D (Intermediate) unfortunately did the Mathe- 
nmatics concepts subtest for a pre-test and thus is not included in this 

section of analysis. 

a. Within proup differences . Pre-test and post-test gains were 
■ analyzed for both USMES and Control groups and for all the individual class- 
rooms. Means (pre-test, post-test, and adjusted) are presented in Table 33. 
An examination of the pre-test and post-test means indicates that, except 
for the Control class of pair E (an advanced grade level), positive gains 
were realized. Four of the five USMES classes showed statistically signifi- 
cant gains from pre-test to post-test, while one USMES class showed no 
statistically significant improvement. Of the Control classes two gained 
significantly, and three showed no significant gain. Repeated measures 
analysis of variance tables for each class can be found in Tables 34 through 
38. The range of gains for the USMES classes were from a low of .07 (USMES 
class A/Primary) to a high of 18 (USMES class B/Intermediat^e) , and for the 
Control classes, from -1 (Control class E/Advanced) to 14 (Control class A/ 
Primary) • . 
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TABLE 33 



Means and Standard Deviations for Pro-Test 
and Post-Test Mathematics 
Computations 



Pair 


Class 


N 


l^ru-Tcsc 


Post-Test 


Adjusted 


Primary 
A 


USMES 


18 


130.22 
C 7.24) 


130.39 

J J . V £. } 


131.30^ 




Control 




120. 59 
( 6.59) 


1.34.86 
(10.72) 


134.00 


T r> t- o YTTlO H "i f\ t 

X n L e LlUc u i. ca u c- 


" USMES 


22^ 


144.68 
( 8.67) 


162.59 
(10.38) 


163.00 


B 


Control 


21 


146.14 
(13.28) 


149.19 
(14.40) 


148.80 




USMES 


16^ 


L73.81 
(14.14) 


183.62 
(15.97) 


177.70 


c 


Control 


18 


159.17 
(14.92) 


162.67 
(15.47) 


168.60' 


Advanced 


USMES 


19^ 


187.05 
(24.98) 


195.84 
(17.19) 


192.20 


E 


Cent rol 


33 


173.09 
(19.29) 


172.21 
(17.67) 


17 5.90 




USMES 


22" 


196.36 
■ (10.35) 


202.73 
(11.68) 


202.70 


F 


Control 


2^' 


194.92 
(12.55) 


199.67 
(12.i'2) 


199.70 



a. Figures in parentheses arc the standard deviation 

b. Represents post-test mean adjusted for pre-test diEEerences 
c*. Statistically significant at or beyond the .05 level 
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TABLE 34 



/ 



Repeated Measures Analysis of Variance Treatment (USHES, Control) 
by Assessment (Pre-Test, Post-Test) Mathematics 
C..'mputation for Pair A 







Sum of 


Mean 




P 


Variance 


df 


Squares 


Squares 


F-Ratio 


Between Subjects 


39 


11680,00 


299,487 






Treatment (T) 


1 


131.00 


131.000 


0.431 


.522 


Subjects X T 


38 


11549.00 


303.921 






Within Subjects 


40 


14472.00 


361.800 






Assessment (A) 


1 


1255.00 


1255.000 


3.899 


.053 


T X A 


1 


986.00 


986.000 


3.063 ■ 


.085 


Subjects X P X T 


33 


' 12231.00 


321.868 






Total 


79 


26152.00 


331.038 
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TABLE 35 



Repeated Measure 'Analysis of Variance Treatment (USMES, Control) 
by Assessment (Pre-Test, Post- Test) Mathematics 
Computation for Pair B 



Sources of 




Sum of 


Mean 


F-Ratio 


P 


Variance 


df 


Squares 


Squares 


Between Subjects 


42 


9092.000 


216.4?6 






Treatment (T) 


1 


766.000 


766.000 


3.772 


.056 


Subjects X T 




8326.000 


203.073 






Within Subjects 


43 


6816.000 


158.512 






Assessment (A) 


I 


2439.000 


2439.000 


31.339 


.000 


TxA 


■ 1 


1186.000 


1186.000 


15.239 


.000 


Subjects X A X T 


41 


3191.000 


77.829 






Total 


85 


15908.000, 


187.153 







TABLE 36 



Repeated Measures Analysis of Variance Treatment (USMES, Control) 
by Assessment (Pre-Test, Post-Test) Mathematics 
Computation for Pair C 



Sources of 




Sum of 


Mean 


Variance 


df 


Squares 


Squares 


Between Subjects 


33 


18157.00 


550.212 


Treatment (T) 


I 


5369.00 


5369.000 


Subjects X T 


32 


12788.00' 


399.625 


Within Subjects 


34 


2)70.00 


81.471 


Assessment (A) 


1 


712.00 


712.000 


TxA 


I 


169.00 


169.000 


Subjects X A X T 


32 


1889.00 


' 59.031 




AT 




312.343 



F-Ratio 



13,435 



12.061 
2,863 
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TABLE 37 

Repeated Measures Analysis of Variance Treatment (USMES, Control) 
by Assessment (Pretest, Post-Test) Matheir.atics 
Computation for Pair E 



Sources of 
Vaiiancc 




Sum of 
Squares 


Mean 
Squares 


F-Ratio 


P 


Between Subjects 


51 


39903-00 


782.411 






•Treatment (T) 


1 


8519.00 


8519.000 


13.572 


.001 


Subjects X T 


50 


31384.00 


627.680 






Within Subjects 


52 


7800.00 


150.000 






Assessment (A) 


1 


183.00 


183.000 


1.297 


.259 


T X A 


1 


564.00 


564.000 


. 3.998 


.048 


Subjects X P X T 


50 


7053.00 


141.000 






Total 


103 


47703.00 


463.136 
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TABLE 38 



Repeated Measures Analysis of Variance Treatment (USMES, Control) 
by Assessment (Pre~Test, Post-Test) Mathematics 
Computation for Pair F. 



Sources of 




Sum of 


Mean 








df 


Sauares 


Sfiiiares 


F-Ratio 


■ P 


Between Subjects 


45 


9590.00 


213.111 






Treatment (T) 


1 


118.00 


118.000 


0.548 


.530 


' Subjects X T 


44 


9472.00 


215.273 






Within Subjects 


46 


3361.00 


■ 73.065 






Assessment (A) 


I 


701.00 


701.000 


11.657 


.002 


Tx A 


1 


14,00 


14,000 


0.233 


.637 


Subjects X P X T 


44 


2646.00 


60.136 






Total 


91 


12951.00 


142.319 
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Tables 39 and 40 are summaries' of the repeated measures analysis of 
variance for Mathematics Computation. Overall, both groups (USMES and 
Control) showed statistically significant growth over the pre-test/post- 
test period. The USMES sample gained 8.88 scaled score points and the 
"Control Sample gained 5*85 points. 

In summary, all the USMES classes gained in Mathematics Computation 
skills, over the pre-tes t/pos t- tes t assessment period. 

b. Between f^toup differences . As with the Reading Comprehension, 
the comparison of the USMES and Control classes was done with a two factor 
analysis of covariance. 

The first factor in the analysis was (USMES vs. non-USMES) and the 
second factor was Grade Level (Primary, Intermediate, and Advanced). Table 
41 contains the combined pre-test, post-test, and adjusted means for each 
factor, while Table 42 presents the summary of the analysis of covariance. 

After adjustment for pre-test differences, there was a significant differ- 
ence between the USMES and Control classes in Mathematics Computation, a 
significant difference among the Grade levels, and also a significant inter- 
action between Treatment and Grade level. An examination of Figure 6 shows 
that at the Primary level the Control group surpassed the USMES group by 
approximately 10.9 points, while at the Intermediate and Advanced levels 
the USMES group surpassed the Control group by 12.8 and 9.3 points, respec- 
tively. 

Table 43 presents a summary of the analysis of covariance for each in- 
dividual class. In pairs B and C (Intermediate) and 5 (Advanced) signifi- 
cant differences in favor of the USMES classes were found. In all other 
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TABLE 39 



One-Way Repeated Measures Analysis of Variance 
for Mathematics Computation-USMES 



Sources of 




Sum of 


Mean 




Variance 


df 


Squares 


Squares 


F-Ratio P 


Within Subjects 


108 


163972.00 


1708.04 




. Assessment (A) 


1 


3820.00 


3820.06 


21.583 .001 


Error • 


108 


16991.00 


176.99 




Total 


217 


184783.00 







TABLE 40 
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One-Way Repeated Measures Analysis of Variance 
for Mathematics Computation-Control 



Sources of 




Sum of 


Mean 


F-Ratio 


P 


Variance 


df 


Squares 


Squares 


Within Subjects ' 


143 


175038.00 


1224.04 






Assessment (A) 


. r' 


2466.00 


2466.00 


22.697 


.00005 


Error 


■143 


15537.00 


108.65 






Total 


287 


193041.00 









I 

H 

to 

I 
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TABLE 41 

Combined Pre-Test, Post-Test and Adjusted Means for Treatment 
(USMES and Control) and Grade Level (Primary, Secondary, 
and Advanced) Mathematics Computation 



Group 


N 


Pre-Test 


Post-Test 


Adjusted 


USMES 


97 


166.8 


17 5.7 ' 


164.4 


Control 


118 


160.8 


165.3 


160.7 


Primary 


40 


124.9 


132.8 


152.7 


Intermediate 


77 


154.2 


163.3 


164.2 


Advanced 


98 


186.4 


190.4 


170.8 
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TABLE 42 



Two-factor Analysis of Govariance for Mathematics Computation Treatment 
(USMES vs. Control) by Grade Level (Primary, Intermediate, Advanced) 



Sources of 




Sura of 


Mean 






Variance 


df 


Squares 


Squares 


F-Ratio 


P 


Treatment (T) 


1 


2136.348 


2136.348 


9.1950 


.0028 


Grade (G) 


2 


2953.344 


1476.672 


6,3550 


.002 


Tx G 


■ 2 


3937.800 


1968.900 


8.4740 


.0003 


Error 


246 


. 57157.608 


232.348 






Total 


251 


66185.100 


263.686 







TABLE 43 



One-Way Analyses of Covariance by Glass 
for the Mathematics Computation Subtest 



... 

Pair 


Sources of 
Variance 


df 


Sum of 
Squares 


Mean 
Squares 


F-Ratio 


P 


Prr.nary Level 


Treatment 


1 


46,88 


46.88 


0.079 


.780 


A 
n 


UK LUl 


37 

•J 1 


21914»40 


592.28 








Total 


38 


21961.28 


577,93 






Intermediate Level 


Treatment 


1 


2137.61 


2137.61 


16.718 


.0003 


B 


Error 


40 


5114.60 


127.86 








Total 


41 


7252.21 


176.88 








Treatment 


1 


563.07 


563.07 


4.972 


.033 


C 


Error 


31 


.3510.87 


113.25 








Total 


32 


4073.95 


127.31 






Advanced Level 


Treatment 


1 


2901.45 


2901.45 


15.980 


.0003 


E 


Error 


49 


8896.83 


. 181.57 








Total 


50 


11798.28 


235.97 








Treatment 


1 




56.22 


0.569 


.455 


F 


Error 


43 




98.86 








Total 


44 
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Figure 6»--Adjusted Cell Means by treatment and grade 
factors on the SAT Mathematics Computation 
subtest. 
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classes, there were no statistically significant differences. However, 
an examination of the adjusted means shown in Table 43 indicate that the 
Control classes of pair A perform better than their USMES counterparts in 
Mathematics Computation after adjustment for pre-test differences. 

In Mathematics Computation, the USMES classes performed better than 
their Control counterparts at all grade levels, except Primary. As was 
the case with Reading, the exposure does not hinder, and in fact, beyond 
the Primary Grade level, may even enhance learning in the basic areas of 
Mathematics Computation. 

Chapter Sununary 

In response to interview questions about the effects of USMES on stu- 
dent performance, teachers cited many favorable outcomes' of USMES. They 
felt that as a result of using USMES, students had become more responsible 
for their own learning. The teachers also cited student growth in data 
collection abilities, graphing, hypothesis testing, decision making and 
verbal conununication amongst peers as decided strengths of the program. 
Most felt that their students had become more Inquisitive, more logical in 
their thinking, and more self-reliant. 

Teachers also cited improvement in some of the basic skills areas, most 
notably, arithmetic applications, and language arts, but these replies were 
clearly dependent on the unit challenge which teachers used in their USMES 
classes. USMFS teachers who were- familiar with a variety of units stressed 
this qualification. 

Primary grade teachers complained that their students encountered dif- 
ficulty with some units (usually the Describing People challenge) when the 
children needed to organize data which they had collected. These younger 
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children were frustrated in their progress on the unit because they had 
not acquired some of the basic arithmetic skills needed to assemble or 
make sense out of their data. 

Teacher perceptions of their USMES students' development in problem 
solving were not supported by the analysis of the Playground Problem Test 
results. There are many possible explanations for this discrepancy. Some 
are more probably than others. 

One night question the validity of the teachers' perceptions or the 
honesty of their responses during the on-site interviews. While such criti- 
cisms cannot be dismissed entirely, the very favorable teacher responses • 
regarding the effects of USMES on their students^ problem solving behaviors 
were consistent across interviewers, across geographical areas, across grade 
levels, across units, across school community socioeconomic levels. Further- 
more, teachers were discriminating in their responses during the interviews 
as a whole. Many qualified their observations about student effects of 
USMES, as noned earlier in this report. Moreover, they offered very criti- 
cal appraisals on other issues: teacher training, support networks, mater- 
ials, and resources. 

More likely areas for explanation of the discrepancy between teacher 
perceptions and student performance data for problem solving lie with the 
unsatisfactory testing procedures and with the instrument itself. The gen- 
eral problems with 1973-74 data collection cited earlier in this report 
applied to the collection of both basic skills data and problem solving data. 
However additional questions must be raised about the value of the Playground 
Problem data because, unlike the SAT, the Playground Problem requires special 
training for proper test administration. Without rigorous training, and in 
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the absence of a precise written guide to test administration (the Manual 
in Appendix A was not prepared until after the 1973-74 data collection 
period), some observers invalidated the responses of some groups by using 
unacceptable testing procedures. (See .^Sec^i^ti[ V of^ the scoring protocol 
for a listing of these problems.) 

Even those results which were not invalidated by unacceptable testing 
procedures are suspect. The reliability and validity of the test itself 
needed to be established. (The review of problem solving research and 
measurement and the development and refinement of new problem solving mea- 
sures are a substantial protion of the work still underway on the 1974-75 
USMES evaluation. Information on the reliability and validity of the Play- 
ground Problem, beyond the scorer reliability procedures and the content 
validation process already achieved, will be given in the 1974-75 report.) 

Results of the analyses of basic skills data were favorable toward 
USMES. Overall, both USMES and Control classes showed a significant gain 
in SAT Paragraph Meaning scores from pre-test to post-test administration. 
Examination of gains by classes revealed that all USMES sample classes and 
five of the six Control classes realized significant increases. 

Analysis of overall differences between USMES and Control groups in 
these Paragraph Meaning scores, after adjustments were made for pre-test 
differences revealed that USMES groups attained significantly higher scores 
than the Controls. Significant differences in adjusted post-test means were 
also observed among grade levels. As one would expect the advanced classes 
were superior to intermediate classes,, which, in turn, were superior to 
Primary classes. 1 T Q 
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Analyses of the Mathematics Computation SAT subtest scores showed that 
as a whole, both USMES and control classes gained significantly from pre- 
test to post-test administration. After adjustments for pre-test differ- 
ences in these scores were made, the analysis revealed that overall, the 
USMES groups were superior to Control groups in post-test performance in 
Mathematics Computation. However, this overall result must be qualified 
by grade level. Control groups were superior to USMES groups at the pri- 
mary level. Comparisons at the intermediate and advanced grade levels of 
elementary school classes in the sampl'e showed USMES groups super7,or to 
Control groups. This treatment by grade Ifevel interaction is corroborated 
by the report of teacher perceptions of their students^ development in 
arithmetic computation skills. 

Limitations to the student performance data necessarily restrict the 
confidence one can attach to these results. Larger, more representative 
samples, and careful data collection with valid instruments are goals for 
the 1974-75 evaluation. Teacher insights will continue to be used as an 
important source of information about USMES. 
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CHAPTER V 



TEACHERS' APPRAISALS OF USMES MATERIALS 

Teachers' appraisals of USMES materials were solicited through the 
fol lowing statement: 

Often times the success of a new program involves 
the development of~U'S6ful7"'appropriate materials, 
''7" tools, and references. We'd like to know which USMES 

materials are...useful to you$ which are not, and if 
there are any new materials which you think should 
be developed to assist students and/or teachers. 

Interviewers were directed to give the respondents ample time to cotnment. 

Then they used the following probes as they seemed appropriate: 

You didn't mention the: 

(a) Design Lab 

(b) Supplies '.or the Design Lab 

(c) Teacher resource manual (s) 

(d) Technical papers 

(e) How-to-cards 

(f) Audio versions of How- to-cards. 

If necessary, the interviewers solicited additional comments with these 
further probes: 

Which ones (in each category) did you use? 
For what purposes? 

The following evaluation of USMES materials iii based on the responses 
given by the eighty USMES teachers to the interview described above. The 
order we have chosen in which to discuss these materials reflects our per- 
ception of the relative importance assigned to them by the teachers. 

The Desipn Lab and Its Supplies 

There can be no question that the Design Lab was seen by the teachers 
as a vital part of the USMES project. In fact, many teachers (and most of 
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the principals whom we interviewed separately) stated that getting a 
Design Lab in their schools was a major motivating factor for undertaking 
the USMES program. It seemed paradoxical then, that the Design Lab also 
appeared to be one of the strongest sources of discontent with USMES. The 
discontent stemmed from various problems: space, staffing, and supplies. 

A. Space for the Design Lab 

In many schools, there was physically no room for a separate Design 
Lab, resulting in one of the following three situations: 

(a) The Design Lab was located in one of the teacher's 
classrooms where it could be used easily by that teacher, 
but only with great difficulty by anyone else; or 

(b) The Design Lab was put into another school in that district, 
and the children were bussed infrequently to use it; or 

(c) There was no pretense of having a Design Lab. 

It would be difficult to assess which of these situations irritated the 
teachers the most. 

B. Staffing for the Design Lab 

Where a Design Lab did ejcist as a facility separate from the classroom, 
its use was limited by lack of staffing. One constant theme echoed by vir- 
tually all of the teachers whom we interviewed was the need to have the De- 
sign Lab staffed on a regular basis so that it would be available for the ' 
children's use when the need arose. 

"Assigning a class to the lab on Friday from 1:15 to 1:45 defeats its 
purpose. 

"Trying to work with 30 children in the Lab is equally unsatisfactory." 
The teachers noted that both of these typical restrictions were necessary 
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if the only people available to staff the Design Lab were the classroom 
teachers themselves. The only appreciable variability among teachers on 
the issue of staffing the Design Lab arose in their suggestions as to who 
should staff the lab: student-teacher, parent, para-professional, teacher 
aide, release-time teacher, mature high school students, retired skilled 
tradesman. In those few schools where the Design Labs were staffed on a 
regular basis (in one case, very successfully by a retired Army Sargeant) , 
the success of the Lab was highly evident, and the teachers were clearly 
enthusiastic about the merits of the USMES program. 
C. Supplies for the Design Lab 

Comp?r.ints about the unavailability of supplies for the Design Lab were 
widespreao among the teachers whom we interviewed. The reasons offered for 
this lack of materials were diverse: 

(a) Some teachers reported thait no money was available to pur- 
chase the supplies. Other teachers claimed that adminis- 

' trators were unwilling to spend money which had been ap- 
propriated for this purpose. Still others complained about 
lost requisitions or other undue delays in securring mater- 
ials for the labs. 

(b) A less common accounting for the unavailability of supplies 
for the Design Lab came from teachers in a handful of inner 
city schools. Here the materials were purchased but were 
subsequently stolen. Very little remained of the original 
purchases in these schools. • (In one of these schools, the 
principal resorted to locking the remaining tools and mater 
ial s in his private lavatory to prevent further theft. 
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Needless to say, these materials were not readily avail- 
able to students.) 
(c) In some schools, the teachers reported that their USMES 
classes had to compete with those of non-USMES teachers 
for use of the Design Lab and its supplies. The prevail- 
ing viewpoint among principals and teachers in such schools, 
as reported by the USMES teacher interviewees, was that 
the program should not be sustained for just a few classes, 
but that the entire school should have access to the Design 
Lab. Consequently, the materials were consumed quickly. 
These expressions of discontent with the arrangements for locating, 
staffing, and supplying the Design Lab, although widespread, varied con- 
siderably in intensity. Those teachers who were eager to continue using 
USMES expressed confidence in circumventing or dealing with the problems. 
For example, many teachers and their principals bypassed the problem of in- 
adequate or unavailable supplies for the Design Lab by soliciting tools, wood, 
batteries, or other materials from parents and/or local businessmen. In 
some cases, parents volunteered materials. In still another school, the 
USMES classes conducted bake sales to raise money for the Design Lab mater- 
ials. 

Many of the teachers in new USMES schools were advised, or decided them- 
selves, to wait until the Design Lab was available and equipped before under- 
taking an USMES unit. These teachers and their principals alike cited the 
unavailability of the Design Lab as the reason they did not use the program 
sooner after their training. In some cases, the rationale seemed sincere; 
in many others it seemed like an excuse to delay getting involved in USMES. 
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Several teachers noted that the USMES program did not need to revolve 
around the Design Lab--in fact many units did not require its use at all. 
These individuals volunteered the opinion that many colleagues in their 
locales delayed implementing USMES unnecessarily because the developers 
oversold the Design Lab. Apparently, the Design Lab did sell the program 
but teachers blamed the developers for emphasizing the Design Lab's uses 
at the expense of the program's philosophy and its possible applications 
without Design Lab facilities. 

Clearly these Design Lab problems exist in varying degrees in the vast 
majority of the sites where the USMES program is being implemented. However, 
these problems are administrative, not conceptual; it would be grossly un- 
fair to overlook the favorable aspects of the Design Lab. From the teachers' 
viewpoints the motivating influence of the Design Lab on children was e- 
normous. Teachers reported that the students fortunate enough to have ac- 
cess to a Design Lab loved using it. Those in the program looked forward 
to going to the lab; those not in the program constantly inquired about it. 

The seeming paradox — that one of the most desirable features of the USMES 
program, the Design Lab, was also one of the greatest sources of discontent 
about the program--"is not so paradoxical in light of the foregoing discussion. 
Teachers' felt that the Design Lab could be a very powerfuU learning oppor- 
tunity. They were vocal about their frustrations in not being able to use 
the Design Lab at all, or in not being able to use it to its best advantage. 

We will preface the discussion of the rest of the USMES materials with 
a description which typified the response we received about the materials 
in the USMES program. Teachers talked readily and at length about: the De- 
sign Lab. When we began to probe for responses about other materials, it 
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was not unusual to get a blank look. After we specifically asked the tea- 
chers to comment on the Resource Manuals, or How- to-cards , or technical 
papers, we found it necessary to describe these materials at length in order 
to help the teachers recall the item we were talking about* We can think 
of no other description which describes more aptly and succinctly the tea- 
chers* feelings about the degree of usage and utility of these materials. 

Teacher Resource Manuals 

In order of importance to teachers, the Resource Manuals follow the De- 
sign Lab, but at a considerable distance. When we asked teachers to comment 
frankly on how helpful the Resource Manuals were to them for the units they 
were using, the vast majority of the teachers said that the Manuals were of 
very limited value, if in fact they had used them at all. 

Teachers' complaints about the Resource Manuals encompassed a number of 
different kinds of criticisms, and they revealed a diversity of expectations 
for the Manuals and a diversity of viewpoints on the USMES program. 

One kind of criticism came from teachers who said they never referred to 
the Manuals because they were too directive for the intended philosophy of 
the USMES program. Such teachers contended that a challenge should not he 
contrived, that it must flow from the students, and that those problems which 
arose naturally in class were not likely to appear in a Resource Manual for 
a given unit. Therefore, in the viewpoint of these teachers, the Manuals and 
indeed the units were useless. These teachers seemed to be rejecting the 
Manuals, even in concept. 

Another large group of teachers concurred that the Manuals were not useful 
to them, but for a somewhat different reason. These teachers noted that 
unless one had gone through a unit in a workshop or with another teacher, the 



-141- 



Manuals were of little help. Furthermore, these teachers explained, if one 
had pursued the training, then one did not need the Manuals. 

A third kind of opinion about the Manuals reflected criticism of the 
Manuals themselves, but not the concept of the Manuals. These several 
teachers pointed, out that one could_not use the USMES program simply by 
reading the Manuals but that the Manuals could be helpful to teachers after 
they had been through one unit at a workshop. These teachers criticized 
the Manuals for being "too wordy," "too padded with case studies," or "not 
concrete enough." The teachers wanted a short, concise, "one, two, three" 
approach to a unit, with a brief overview of what to expect, and specific 
suggestions on how they should present "the challenge." 

We heard very few favorable comments on the Manuals. These came from 
teachers who said they appreciated the flowcharts or diagrams of possible 
directions the unit could take, and from teachers who said they were able 
to use a few ideas about activities that worked in other classes. 

In summary, then, the Resource Manuals were not very helpful to USMES 
teachers. The teachers preferred to learn how to use a unit at a workshop 
or from other teachers. They constantly reminded us that they "just didn't 
have time to read all that." (Another pervasive comment from Development 
Teachers is relevant here. During earlier portions of our interviews, these 
teachers expressed great resentment at having to compile logs of their USMES 
developmental activities in their classrooms for the USMES Central Staff. 
It is these edited log materials which comprise the case studies in • 
Resource Manuals which other teachers objected to most strenuously.) 
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''How To' ' Cards (Audio and Written Versions) 

It is easiest to appraise the degree of usage of the audio "How To" 
cards first, because very few teachers had heard of their existence and 
thus had nothing to say about them. Teachers from only two geographical 
areas had any comments about these aduio tapes. Those teachers whom we 
interviewed in one large, urban school district expressed resentment that 
their school administrators would not, or had not, provided funds to pur- 
chase the audio "How To" cards. These teachers felt that the audio tapes 
would be very helpful for those of their students who had difficulty read- 
ing. Yet, thes. teachers had never reviewed the audio tapes. 

One teacher in a rural school had tried using the tapes with his seventh- 
grade class which he said was comprised "mostly of slow-learning boys." 
This teacher reported that his students felt the tapes were "condescending." 
He noted, however, that many of his students were older than the average 
seventh grader. 

With these two exceptions, the teachers whom we interviewed were either 
unaware that audio "How To" cards existed, or they vaguely remembered that 
they had heard of them but expressed no interest in using them. 

The written "How To" cards seemed to be available at all of the USMES 
schools which we visited. In many c£.ses, they were in evidence in the De- 
sign Lab, posted on its walls, or on the carts which were designed to serve 
as movable Design Labs. Despite this advertising, however, the vast majc .ity 
of teachers acknowledged that their students used them rarely, if at all. A 
particular complaint made by lower grade teachers was that the reading level 
was too difficult. As these teachers repeatedly pointed out, one set of 
cards could not possibly service all levels of children in grades one through 
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eight. One primary grade teacher was attracted by the concept of the How- 
To cards, i.e., to encourage children to resolve their own difficulties, 
to answer their own questions, to get help on an individual basis when it 
was needed. This teacher had begun to develop her own »'How-To'' cards u- 
sing a comic book frame approach incorporating limited verbal materials 
with pictorial representation. 

Upper grade teachers as well stated that their students did not use 
these cards. Several teachers admitted to using the cards themselves as 
•a resource for teachers. Most agreed that it was easier in any case to 
show or tell their students how to. perform a certain task. Several tcacheriv 
commented further that their students were not accustomed to learnivig. by^/; 
the "How To" card approach. When we asked some of the teachers if they, 
had offered group instruction in the. use of the cards, only one teacher 
reported that she had, and even then, that her effort did not result in in- 
creased usage of the cards. 

In summary, the How-To cards have not been successful, if the amount- 
of -usage and teacher enthusiasm are indicators of success. Lower grade 
teachers categorically rejected the utility of the. cards because of their 
difficult reading level. Upper grade teachers were also negative about the 
utility cf the cards, but the reasons for their negative aporaisal were less 
clear. It may be that the cards were too difficult even for mv-ny upper 
grade students. Possibly the teachers preferred to be more directive with 
their students in the areas of instruction offered by the How-To cards. 
Many teachers seemed to be attracted to the concept of encouraging students 
to learn needed skills and acquire facts on their own, but in practice the 
teachers were not using the ca1?ds in this way. 
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Technical Papers 

Least used of all the USMES materials and resources were the technical 
papers. The vast majority of the teachers admitted that the papers were 
too difficult for them to understand. The few who said the papers were not 
difficult explained that they had no time to ■ r.hem. There was not one 
positive indication that the technical papers ■•■<■ a useful contribution to 
The USMES program. 
Chapter Summary 

Teachers' reports of tl.o degree of usage of USMES materials and their 
appraisals of the effectiveness of these materials indicate that consiSerable 
attention should be devoted to revising and/or eliminating the Resource Man- 
uals, the "How To" cards and the Technical Papers. Clarity, simplicity and 
conciseness would be attributes in these materials valued by the teachers. 
The sheer volume of the materials in their present form discouraged many 
teachers from using them. 

Clearly, the Design Lab is by far the most attractive tangible component 
of the USMES program for the teachers. Many difficulties have arisen with 
supplying, staffing, and making the Design Lab accessible to teachers and 
their students. ^Nevertheless, those teachers who are committed to the USMES 
program, and especially those who are not isolated in schools without other 
USMES teachers, have been resourceful in sustaining and improving the Lab's 
uses. Future USMES workshops at the national level and - the local level 
should include presentations from experienced USMES teachers on suggestions 
for coping with the problems of scheduling, staffing, and supplying the De- . 
sign Lab for effective ''tudent use. 
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CHAPTER VI 

THE EFFECTIVENESS OF USMES TEACHER TRAINING 



Background 

The USMES approach to learning the process of real problem solving re- 
quires that the students themselves, not the teacher, discover and analyze 
a problem meaningful to them, choose the variables that should be investi- 
gated, collect and analyze the pertinent data, judge the correctness of 
their hypotheses and form appropriate conclusions and recommendations. 
The teachers' role in this process is that of a coordinator or a collabor- 
ator rather than the more conventional role of a director or an informa- 
tion-giver. The inexperience and unf amiliarity of many teachers with this 
USMES-styled role compounds the importance of effective teacher preparation 
in the USMES project. 

Various teacher training models have been used by the USMES project. 
For several years. National Summer Workshops have been held to train tea- 
chers, who were new to USMES, in the implementation of first trial units, 
and to afford experienced USMES teachers the opportunity to discuss plans 
■ for the development of new USMES units. Participants at these National Work 
shops represented an extensive range of geographic areas. 

The USMES project also implemented two models of district-:wide teacher 
training programs which were more intensive by geography. The Lansing Dis- 
trict Implementation Experiment involved University-district collaboration 
in a medium-sized community. The Chicago Distric t Implementation Experi- 
ment involved only the district resources found in a very large city. 

These inservice training efforts have been complemented by pre-service 
training components for students in schools of education. 
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AU of the above models have relied on the more immediate impact of 
an experienced USmES training staff for preparation of teachers to utilize 
UaiES in their classrooms. The basic strategy was limited in that it pro- 
duced only "first generation" USMES users; none of these models were based 
on pyramidal structures. 

• Realizing that widespread implementation of the USMES program could 
not be achieved efficiently with training models limited to first genera- 
tion effects, the USmES Central Staff adopted a training strategy with 
built-in multiplication factors for its limited experienced staff and 
limited financial resources. This strategy, patterned after the generally 
successful technique used by many NSF projects, involves training districr. 
resource teams who, in turn, train and support district teachers. 

The USMES Central Staff invited each of approximately 25 districts to 
plan, with project staff assistance, a method of implementation that suited 
the districts-needs and resources. "The. size and composition of the team 
and the design of the school-year implementation program were negotiated 
separately with each interested district to optimize the rate and quality 
of implementation." Only districts committed to plans that had the pos- 
sibility of a large number of classroom implementations were invited to 
be part of the Resource Personnel Workshop in the summer of 1973. 

The following 17 resource teams attended this two-week RPW training 
session in Lansing, Michigan: 

(a) From California-- 

Bakersfield City School District Team 
Bakersfield Greenfield School District Team 
Bakersfield Standard School District Team 
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Carmel Team 

Fuller ton Team 

Los Angeles Del Amo Group 

Los Angeles 95th Street School Group 

Los Angeles Valley Group 

Monterey Team 

(b) From Georgia — 
Atlanta Team 
La Grange Team 

(c) From Illinois-- 
Chicago Team 

(d) From Massachuse tts-- 
Arlington Team 

(e) Representing centers for the Elementary Intern Program 
affiliated with Michigan State University — 

Detroit Area Team 
Flint Area Team 
Lansing Area Team 

At the Michigan workshops, the project staff trained over 90 RPW .par- 
ticipants in the philosophy and content of USMES units and in teacher 
training strategies. Documentation of the workshop format and activities 
as well as an account of participants* immediate evaluation of the RPW and 
their initial plans for local training are contained in the '^Report of the 
Resources Personnel Workshop, Lansing", Michigan, June 24-July 5, 1973,** 
prepared by the USMES Central Staff at EDC. 
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Purpose 

The purpose of this chapter is to provide formative feedback on the 
effectiveness of USMES teacher training to the National -Science Foundation, 
to the USMES Central Staff and their consultant/specialists for training, 
and to existing and prospective USMES District Resource Teams. Special at- 
tention is devoted to the effectiveness of the 1973-74 Resource Personnel 
Workshop efforts, but more generally, pervasive issues about USMES teacher 
training are also addressed. Recommendations from the USMES Evaluation 
Staff for future applications of various USMES training models are presented. 

Most of these results have already been presented to the USMES Central 
Staff and discussed with them as soon as substantial trends could be noted 
and patterns established. (The sources of information, especially of neg- 
ative criticisms were not identified to the developers, however.) The Cen- 
tral Staff incorporated sou.^^ of our recommendations in subsequent training 
efforts. Given the especially formative nature of this part of the total 
1973-74 USMES Evaluation Project, noteworthy reactions of the Central Staff 
to the feedback on USMES teacher training are included along with the re- 
sults themselves. 

This assessment of USMES teacher training is based upon interview data 
from the following respondents: 

(a) 26 development teachers 

(b) 16 implementation teachers 

(c) 16 ^^second generation,^' local ly- trained RPW teachers 

(d) 20 Chicago Workshop trained teachers 

(e) 2 pre-service trained tealchers 

(f) 13 RPW team leaders 
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(g) 17 e;.ernentary school principals 

(h) 3 district level curriculum supervisors 

(i) 5 district superintnedents or associate superintendents. 
The respondents and the interview method were described in Chapter 11. In- 
terview guides for tne largest respondent groups-^USME S teachers, princi- 
pals, and team leaders, are shown in Appendices A, B, and C, respectively. 

Resul ts 

Naturally, teachers, team leaders, and administrators emphasized differ- 
ent kinds of issues about USMES teacher training because their perspectives 
and role responsibilities were different. The interview responses below, 
therefore, are organized and discussed first by respondent-roles, and then 
by a synthesis of responses across groups. A summary of our obsen/ations 
follow. 

A. Teachers' Viewpoints on USMES Teacher Training 

Our teacher respondents were about evenly split between those who had 
attended national workshops (26 development teachers and 16 implementation 
teachers), and those whose training had taken place in -.heir local school 
districts (20 Chicago workshop trained teachers; 16 "second-generation," 
RPW-trained teachers; and 2 pre-service trained teachers). 

Actually, the location of the training did seem to influence the degree 
of involvement of teachers in the USMES program. The chance to travel to 
a National Workshop was viewed by many teachers as an attractive incentive 
for undergoing USMES training. Conversely, having to pursue training at 
the local level was viewed by many teachers as a penalty, especially after 
some of their colleagues had talked about the extra curricular advantages 
•of the National workshops, particularly those in Boston. 
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Financial incentives also attracted teachers to the Program. The money 
paid to both development and implementation teachers to attend national 
workshops during Summer, 1973 and the money paid to development teacherk 
to prepare logs of their new unit development efforts were seen as impor- 
tant reasons to initiate or continue USMES involvement. In Cfr'Ltornia, 
where teachers must accumulate salary credits to achieve a higher pay: scale 
level, local USMES training workshops were given for credit and were there- 
fore taken. 

We point out these teacher reactions because teachers, preferring to 
see themselves as altruistic, seemed embarrased to admit that money or 
travel played any part in their decision to use USMES. While many teachers 
mentioned money and/or travel, most of those who did tried to "laugh it 
off" and to say they were not speaking in a serious vein. Only two of the 
teachers, one who was no longer using USMES and the other who was soon to 
discontinue its use (he wouldn't g-t paid to do logs so it didn't seem 
worth his while), admitted that mc • was the greatest incentive, to them, 
and, in their judgement, to other teachers as well. 

Aside from the initial appeal of travel or financial incentives, most 
of the teachers volunteered for USMES training. The reasons they gave for 
getting involved in the program were variations on, either of two themes: 
(a) "I was bored with what 1 was doing and wanted to try something new; 
USMES sounded like an exciting way to teach," or (b) "I always taught the 
USMES way but it seems more acceptable to other people if it's part of a 
program." The first kind of comment was offered the more frequently as 
the reason for getting involved in USMES. In many cases, it seemed to be 
a very sincere reason. 
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None of the teachers said that he/she was required to use USMES. In 
fact, all respondents of whatever position or role were unanimous in 
stating that involvement with USMES must be on a volunteer basis. Yet, 
interestingly enough, one school required teachers to try at least one 
USMES-styled unit during the year, and the ^..ogram seemed to b; soing ex- 
ceptionally well there. There, the teachers were enthusiastic about the 
program, and interes n USMES was on the increase. This extraordinary 
result, though was probably due to an extraordinary principal. 

Teachers' answers to the question of whether or not the training met 
their expectations reflected at once the fact that they arrived at the 
workshops with varying levels of information, misinformation, and prepara- 
tion for training. For teachers with no knowledge about USMES workshops 
tended to be confusing. For those with some knowledge of USMES, and es- 
pecially for those with a bent for math or science, the same workshops 
tended to be judged as time-wasting and boring. As one teacher put it: 
"One local workshop combined the interested, ignorant, friendly and hos- 
tile without planning." This kind of mixed' assessment pertained to. national 
workshops Just as frequently as to local training efforts. Consistent with 
this result was the finding that about half of the teachers who attended 
USMES workshops did hot feel sufficiently prepared to use USMF,S in the 
classroom, .while the other half thought their training was beneficial to 
the point of enabling them to proceed to classroom application. 

While teachers' expectations and appraisals of workshops were diverse, 
the suggestions they offered for improving the training recurred with much 
regularity and consistency across sites and across interviewers. These 
suggestions are discussed below, but no priority should be attached to . 
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order of presentation. 

(a) The person who directs the training session was a key 

factor in each teacher's evaluation of the workshop he/ 
she attended. We observed widespread resentment among 
teachers toward the attitudes of the "college people," 
whom the teachers feit were ill-prepared, uninterested, 
and therefore condescending in their treatment of the 
teacher-participants. A few of the college prefessors 
who served as instructors/consultants/specialists at the 
national workshops received repeated praise, but we heard 
the same few names over and over again. The teachers 
cited their qualities of being knowledgable , supportive, 
enthusiastic, dynamic, and familiar with teachers' prob- 
lems as the reasons these few individuals were so effec- 
tive as trainers for USMES. 

Regardless of their training site, teachers commented 
that they would like to see the training sessions incor- 
porate more informal discussion with experienced, believ- 
able USMES teachers. They seemed to feel that they profit, 
ed much more from sharing experiences about teaching USMES 
among themselves than from listening to professors or 
other consultants whose experiences in elementary school 
classrooms were remote or nil. The teachers perceived the 
opportunity to interact with their colleagues about USMES 
as important reason for conducting a workshop, es- 

pecially a national workshop. 
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Many teachers offered the comment that the use of chil- 
dren in workshops was one of the worst aspects of the 
training. The teachers felt that the situations were 
contrived and bore no resemblance to those found in a 
real classroom. However , . most of>-these teachers commented 
further that the basic training technique of showing chil- 
dren 'Using the materials or working on a unit challenge 
was a good one. For national workshops conducted during 
the summer months, they suggested using video tapes or 
films of the program in use in a classroom. An addition- 
al suggestion for local training was that some of the 
sessions be conducted during the school day when released 
time teachers, undergoing USMES training, could witness 
children using USMES ''for leal" in actual classes or in 

L 

the design lab. 

The teachers repeatedly asked for suggestions as to how 
one teacher could mar^age several small groups in a class 
of 30 to 35 children who are engaging in physically active 
and diverse kinds of efforts. Furthermore, the Design Lab 
was seen as an exceedingly difficult responsibility for 
the teachers when they were working with that many children. 
Many teachers expressed concern for the children's safety 
when such large numbers were using the equipment. (At 
this point, teachers again cited the need for staffing the 
design lab so that they could send small groups of children 
to a supervised design lab to work on a particular task.) 
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In regard to teacher training in the use of Design Labs, 

many teachers who had attended the Lansing or Boston 

workshops during the summer of 1973 advised us that the 

concept of the Design Lab was oversold at these workshops. 

These teachers noted that they themselves had assumed that 

an USMES unit could not be done with out a Design Lab in 

» ■ 

the school. They noted that many of their USMES teacher 
colleagues with lesser experience still held this miscon- 
ception about the role of the Design Lab in USMES. 
Several teachers whom we interviewed suggested that national ^ 
level and local level USMES training sessions should separ- 
ate upper and lower grade teachers because they face dif- 
ferent kinds of problems in using the USMES approach with 
their children. A few further commented that lower grade 
teachers would better focus on a more limited number of 
USMES units which seem to work best with the your»ger children. 
When discussing their use of time at the national Workshops, 
a majority of the teachers thought that intensive training 
on a singlle unit was a mistake, They felt incapable of 
using the other units without some introduction to each one. 
Further, USMES units are to begin with a problem which arises 
naturally in the classrooms. But with so limited a reper- 
toire, the teachers were seldom able to discover a suitable 

problem arising naturally and therefore resorted to -con 

triving a problem which related to the only unit, they could 
deal with comfortably. The teachers seemed to be asking 
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for two- or three-day sessions on each of three or four 
units at a two-week national workshop. 

In general, the teachers called for less talking and more 
active involvement at the workshops. (Descriptors such as 
"boring,'' "time-wasting," "Mickey-Mouse," and "like summer 
camp reunions" were used frequently by teachers to charac- 
terize workshops, more so at the national than at the local 
level.) Despite the fact that the USMES program is more a 
philosophy than a set of materials, the teachers were not 
interested in listening to philosophy. They seemed to want 
a very systematic, Vhow- to-do-it" approach so that when they 
went back to their schools, they would know exactly what to 
do. Most of the teachers would like to go through several 
units in exactly the way the children would be expected to 
go through them. Possibly the underlying philosophy could 
be discussed after the units have been experienced., 

A small minority of the teachers noted that discussions 
of the USMES philosophy were certainly /in order and were en- 
tirely consistent with the whole intent of the program. One 
teacher who seemed to reflect this viewpoint commented: ."I 
expected lesson plans and just got ideas. I expected a rigid 
schedule and got freedom. The workshops should be conducted 
exactly like the philosophy." 

Only a small tiumber of teachers directly requested more time 
for skills sessions during the workshops. However, ,we ob- 
served that much of the uneasiness which teachers expressed 
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about proceeding with their units seemed to be attribut- 
able to teachers* 'iJncertainty about the competencies they 
thought would be required of them. 
^' Team Leaders' Viewpoints on USMES Teacher Training 

It was noted earlier in this report that the Resource Personnel Work- 
shop model for USMES teacher training involved training district resource 
teams who, in turn, were' expected to train and support additional teachers 
in their own districts. Each team was comprised of a leader and two to five 
members. 

If the number of "second-generation,*' locally- trained teachers pursuing 
USMES units in their classrooms were to be used as a criterion for evaluating 
the success of the 1973-74 RPW training efforts, then the RPW approach for 
USMES could only be judged a failure. Local applications of the RPW model 
following the Lansing workshop in July, 1973 through the completion of in- 
terviewing in May, 1974 produced far fewer "second-gf^neration" teachers .than 
the Lansing workshop. At the time of the interviews, some of the teams had 
done nothing beyond the initial planning stages for local training. Others 
had just completed informational sessions for prospective USMES users but 
had not yet begun training. Only about half the teams had proceeded with 
training. 

These observations do not deny the potential for a greater yield of 
locally-trained USMES teachers at some, later date. It did appear to the 
evaluators that most administrators at both the— building and district levels 
wanted to see USMES "pilot-tested-- or '.'modeled" by first-generation teachers 
in their schools before they would encourage or allow training for large- 
scale USMES use. 195 .. .. 
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The comments below are based primarily on our discussions (two by tele- 
phone) with 15 of the 17 team leaders who had attended the Lansing, Michigan 
Resource Personnel Workshop for district resource teams. The viewpoints of 
six RPW team members who went to the Lansing workshop, whom we interviewed 
as first trial implementation teachers, are also considered in this section. 
Again, one should not attach an order of importance to the order of presenta- 
tion of the following points: 

(a) All of the team leaders agreed that local USMES training 
should not be' compulsory. Those leaders who could comment 
from experience noted that a variety of advertising tech- ' 

-niques could be used to attract teachers to local USMES 
workshops:^ posters, news-letters, announcements at faculty 
meetings, and especially, by example^^uf-. successful USMES 
teachers. Notices that USMES workshop attendance could 
be used for credit toward salary increments in California 
were acknowledged as helpful in that state. One team leader 
recommended using a low-keyed information session at the 
teachers' minimum day meeting to recruit. He advised: "Make . 
it the order of business. Don't tack it onto a long meeting . 
filled with other agenda items," 

(b) Support, permission, and/or encouragement from key admin- 
istrative people seemed to be important for getting local 
training started. In a school with considerable autonomy 
and sufficient resources, the key administrator was the 
building principal. In other districts, one or. more cen- 
tral level administrators were reported to control when 
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USMES training could take place and/or how much 
support it would get. Noting a relationship between 
widespread dissemination of the program and political 
consciousness, one team leader lamented that "you must 
invite the right administrative people to the workshop 
(in Lansing) 5 don't slight or offend these people." 
In those districts where no local training had taken 
place prior to the date of interview, the main reason 
given for the delay was the lack of a Design Lab. Most 
leaders would not give a .workshop until the lab had been 
set up. A few leaders alluded to clearing up other school 
business or referred to "working around school vacations" 
as other reasons for a delay in training. 
Local workshops which had been given used a variety of 
time frames: one half-day a week for several weeks on 
the teachers* ninimum days (the days on which students 
were dismissed early and teachers remained for planning, 
conferences, or other work); two full-days on a weekend;, 
one or two hours a week after regular school days for 
several weeks; portions of faculty meetings. Only the 
first of these applied options was considered reasonably 
satisfactory, and then it was used in California .y/here 
the teachers also received salary credit for workshop 
attendance. It was generally agreed that after school 
was the worst possible time to give a workshop, and week- 
end tin?eslots were resented by many teachers. 
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Our request for suggestions as to what times would be 
best for USMES workshop sessions produced a firm consen- 
sus that the workshops should be intensive. Most of the 
team leaders whom we interviewed felt that tihey would or 
did loose teachers' interest if the workshops were spread 
out over weeks or even months. Some specific suggestions 
on when to conduct workshops follow: 

( i) an int:ensive workshop in August, just prior to the 
opening of school, with briefer sessions as needed, 
throughout the year to provide continuing assis- 
tance to teachers; 
( ii) a summer session with each day divided into two parts- 
, working alone with teachers and then trying things 
out on a group of children; 
(iii) a three-day intensive session, after the Design Lab 
is set up, ideally with released-time teachers at 
the beginning of the school year, followed by occa- 
sional minimum day sessions for continued assistance; 
( iv) half -day sessions on the teachers' minimum days, once 
a week over six weeks. 
While there was no uniform preference on when to train, 
there were general points of agreement in all of the sug- 
gestions we heard from team leaders,. They would like to 
proceed with intensive trainings after some '»f irst-genera- 
tion" USMES teach^^rs could demonstrate by example that the 
USMES approach works in their classrooms. 
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Almost all .of the team leaders who had conducted an ini- 
tial information session or who had proceeded to actual 
training thought that their efforts to date were success- 
ful. Most based their judgements on subjective criteria 
such as expressions or perceptions of teacher interest and 
enthusiasm. Only a few team leaders were able to cite 
the more objective measure of success— the number of lo- 
cally trained teachers who were using USMES units in their., 
classes. 

Team leaders echoed the USMES teachers' comments on the 

issue of who should couduct the training. (Some of the 

team leaders themselves conducted portions of local train-: 
ing sessions, while other leaders delegated this responsi- 
bility.) The team leaders noted that successful workshops 
are led by dynamic, knowledgable people. The first day 
must be stimulating, and therefore motivating, and the work^ 
shop should be intensive so that, the momentum is not allow- 
ed to die. On the qualities of tiie trainers, one team 
leader commented: "The presenters should not operate out- ■ 
side their area of competence, and their personalities 
should be dynamic. They must convey interest and involve- 
ment." Another active leader offered this suggestion: 
"Get teachers who are involved )to go over the teaching of 



their units and give a synopsis of what happened, what one 
can expect." . 
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The content of the local workshops did not seem to be an 
area of disagreement among team leaders. They all felt 
that the approach of having teachers go through a unit 
themselves, which was used at the national workshops con- 
ducted by the USMES Central Staff, was the best approach 
for local training too. While most of the team leaders 
emphasized that their' teachers must get "hands-on experi- 
ence," a few cautioned that activity-oriented sessions on 
methodology must be interspersed with discussions of the 
USMES philosophy, to insure that teachers don't perceive 

USMES as . a se.t...Qf ...materials .or.. a .Design Lab.« What.xo.nr..... 

stitutes. an appropriate balance between time spent on 
"how-to-do-it" sessions, and time spent on discussion of 
the USMES philosophy, may be a point of disagreement be- 
tween the teachers and team leaders whom we interviewed. 

< 

Or, it may be that some team leaders were trying to bridge 
a gap, )between expressed philosophy and demonstrated appli- 
cation) , which was perceived by several teachers who attend- 
ed the national Resource Personnel Workshop in Lansing, 
Michigan. 

One RPW team leader offered an additional suggestion 
on what content to include in local workshops. We think 
his suggestion merits special note because it would seem 
to serve a need expressed by many USMES teachers. He~rec^^ 
commended that a couple of sessions in a workshop be de- 
voted to specific content, skills, and technical background 
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which teachers would need to use the '"more difficult," 
science-content-laden units. Using the Burglar Alarm 
unit to illustrate his point, he noted that teachers need 
some instruction in the basics of electricity before they 
can deal , effectively with this unit in their classes. It 
is his opinion that without sufficient background, the 
teachers would use the unit manual to acquire a finite 
amount of material which they feel they needed to get 
across to the children. Lacking any real understanding 
of the content themselves, the teachers would disseminate 
measured amounts of information to their students and de- 
feat the purpose of the USMES program. He rejected the 
value of optional skills sessions because he felt that 
most elementary school teachers would not know what scien- 
tific information or skills they lacked. He dismissed 
the utility of the technical papers for this purpose be- 
cause he was convinced that none ot the teachers would 
read them or was capable of reading them, 
(h) Those RPW team leaders who had completed teacher training 
workshops noted the importance of providing continuing , 
support to teachers. They felt that it was the team leader's 
responsibility to identify resource people who could help 
the teachers, and to keep Design Lab materials in adequate 
supply for the teachers. When teachers encountered problems 
with scheduling Design Lab use, with ordering and/or ac- 
quiring Design Lab tools and materials, or with technical 
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aspects of a unit, they needed itranediate assistance. 
Waiting days or even weeks to resolve the difficulties 
was deemed unacceptable because the teachers and their 
students would loose interest, which, once lost, is 
virtually impossible to recapture. 
C. Principals' Viewpoints on U5MES Teacher Training 

It would appear that a knowledgable (about USMES), supportive princi- - 
pal is a necessary but insufficient condition for success with the USMES 
program in a school. In no school did we find an active, committed cadre 
of USMES teachers on the faculty if the principal knew little and cared less 
about the USMES program. And almost without exception, those principals who 
could discuss the program knowledgably and who would provide logistical 
support and encouragement to USMES teachers were principals who had attended 
USMES workshops at the national or local level. 

Having its principal attend a workshop did not insure that a school would 
be the site of much USMES usage, however. Some principals, despite their 
USMES knowledge, commitment and good intentions, were apparently unabl e to 
provide needed assistance to teachers without support from higher adminis- 
trative levels. A few principals had other -motives for attending USMES work- 
shops: travel; the chance to meet other principals; curisodty. Two princi- 
plas attended a national workshop with the primary intention of procuring 
National Science Foundation funds for their schools. The nature of the 
USMES program was of little concern to them; the perceived prospect of out- 
side funding was of great interest. 

A total of 21 principals, 4 of whom also served as team leaders, were in- 
terviewed about the effects which USMES usage has had on their schools, about 
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the administrative problems they encountered in the adoption of this new 
curriculum, about the support for USMES they were receiving from higher 
level admini.trators or others, and the support they were dispersing to . 
USMES teachers. Though we did not interview the principals about teacher 
training directly, several comments emerged from principals on the issue 
of teacher training. In general, these responses came from the more knowl- 
edgable principals. The poin'ts are suima-rized below in no particular order; 

(a) Teachers need to be receptive to the USMES pWrlosophy in 
order to be successful with the program. Echoing the point 
made by team leaders, the principals agreed that teachers 
should not be compelled to use the program. 

(b) USMES teachers cannot be ispt^r.ed from one another; they 
need continuing peer support as much as they need assis- 
tance from administrators. Enthusiasm and commitment for 
the program are nurtured when teachers can consult with 
one another about ideas for using USMES. Training one or 

'two teachers from an area isolated from other USMES activ- 
ity is a waste of time. 

(c) The principal and appropriate higher level administrators 
need to be warned clearly and directly as to what financial 

. and space commitments a school must make in order to use 

the USMES program. This comment was applied most frequently 
to the Design Lab problems--f inding space for it; supplying 



it, staffing it. 
(d) Those principals who were knowledgable about the program 

commented that it would be very difficult for a teacher to 
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use USMES without his/her principal's moral and logistical 
'support. In the same vein they emphasized that USMES 
teacher training sessions should address the issue of what 
content and basic skills would be developed by each USMES 
unit and/or by the program overall. This information would 
be helpful to principals and teachers alike especially in 
those districts where prospective users have difficulty 
seeing how a diffuse, integrated program like USMES could 
meet specific curriculum requirements of the district. 
The " "principals shared the same views held by USMES teachers 
and team leaders on who should conduct training sessions, 
i.e., dynamic knowledgable people who are familiar with 
the problems of elementary^ school teachers. 
One principal, whose school was the site of much USMES ac- 
tivity, offered the following suggestions for USMES teacher 
training: 

"Look at the teacher as a learnsr; see where she is and 
set expectations from there. Do not expect a perfect unit 
the first time: let each teacher experiment by setting 
one more goal for each unit she tries. Don't bog a teacher 
down with an 'official' USMES unit; rather encourage a 
teacher to solve with the USMES philosophy any problem 
which ar.is,es with a group. Identify those most likely 
to succeed and let them start. Be very supportive to 
peoples trying USMES-styled units; reinforce heavily. Tell 
-. teachers to stay away from USMES materials (the Design 
" Lab)' until the USMES method has been tried and the teacher 
feels comfortable with it; then try another unit with 
materials. " 

Another principal, whose school is the site of sustained, 
intensive USMES usage by a limited number of teachers 
offered a different perspective on USMES teacher training: 
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•'Those involved (those who go to national workshops) 
-carry. the b"' ^ ogram doesn't spread without 

teacher lal workshops and getting in- 

volved, buituui xnservice is not enough. Local training 
is too diluted.*' This opinion is shared by a substantial 
number of principals and tearhers# 

Additional Observations by the Evaluation - 

The findings reported in this section have been culled from our inter- 
views with several respondent groups. Unstructured observations by the 
evaluation team during the February-May, 1974 site visitations and infor- 
mal discussions with several key respondents have also been considered ■ 
here. 

Insufficient, inaccurate, or misdirected communication between the 
USMES developers and the workshop participants accounted for many of the 
problems and much of the dissatisfaction with USMES Training which we ob- 
served during our site visits. In the view of many administrators, team 
leaders, and teachers whom we interviewed, the USMES Central Staff failed 
to clarify the financial, space and time commitments which an individual, 
school, or school district must make to USMES usage and especially to' 
USMES training. 

The first major misconception held by a large number of interview re- 
spondents was in regard to the role of the Design Labs in USMES units. 
Frequently, it was assumed that a unit couldn't be done unless there was 
a Design Lab in the school. Some of our respondents who obse'-ved this 
problem attributed it to the USMES developers' oversell of the Design Lab 
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and its uses as the most attractive features of the USMES program. They 
felt the developers should have emphasized the philosophy, the interdis- . . . . „ 
ciplinary nature, and the real life problem solving aspects of the program. 
As a result, prospective USMES teachers anu principals were preoccupied 
with the problems of finding space for the Design Labs in overcrowded build- 
ings, and of supplying and staffing the labs with already strained finan- 
cial resources. 

The USMES Central Staff conceded the problem and noted that misconcep- 
tions about the role of the Design Lab probably had their roots in the 1971 
summer workshop when the staff /was encouraged to make^ sure participants got 
to use the lab, even if a need. was not identified within the course of the 
unit work. There appeared to be some carry over of this policy to later 
workshops through 1973. However, the USMES developers have since made a 
concei-ted---'ef f ort to correct the miscoTicep'tionnDy~craFiTyrn^^ 
Design Lab in their newer written materials and through staff members* ex- 
planations at workshops. 

The responsibilities of participants at the Lansing Resource Personnel 
Workshop for conducting local training constituted a second major area of 
misunderstanding. A significant number of these Lansing RPW^ participants 
did not realize that they were expected to train other teachers in their 
home districts.- They expected only to use USMES in their own classrooms, 
just as First Trial Implementation teachers had done when they returned from 
national USMES workshops. 

Apprised of the confusion over responsibilities for local training, the 
USMES developers explained that they had negotiated with district level 
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administrators about inviting teams to Lansing to prepare for local train- 
.ing..e£forts., .Howeve.r.,: in many cases, these communications .were not relayed 
downward to the principals and teachers who actually participated at Lansing. 

To remedy this problem, the USMES Central Staff has been communicating about 

local training in ' -actions: first, toward the appropriate level in a 

school district's ' .ration; and second, downward to the actual partici- 
pants. The Central Staff now demands written commitments to pursue local 
USMES training from the proper district official before it will invite a 
team and pay its expenses to a Resource Team Workshop. Furthermore, the 
USMES developers now insist upon receiving team rosters so that they can 
issue information about the workshop to the participants direc.tly. The 
policy of direct contact with local USMES trainers appears to have disadvan- 
tages as well, however. (The evaluation project director and associate di- 
-re-cror-TiT)lred-coTrtri-TTuedT^even-increas^ on USMES Central Staff 

resources by local teams during our 1975 site visits.) The dilemma faced 
by the USMES developers is how to clarify ars?ij>-b;e-approprlatel-y^5upRaiiJiiv^_= 
without nurturing dependence on its limited resources. 

A third major problem with USMES teacher training during 1973-74 involved 
setting unrealistic expectations for local training efforts, or even apply- 
ing the RPW teacher training model prematurely. For several teams at the 
Lansing workshop, it was their first exposure to USMES. The designated 
trainers felt inadequate to train others, because they were newly trained 
themselves. At Lansing, they had received intensive training in one USMES 
unit, but they could not tallT^ut other units.. When these inexperienced 
Lansing workshop participants returned to their home districts, they were 
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encouraged by their administrators to delay local workshops until they 
felt comfortable teaching the program, until Design Labs were set up, and 
until they could demonstrate for their local colleagues that the program 
was workable in their districts. Only in districts where these conditions 
were already met, could one find any locally-trained, -second-generation" 
USMES . ■ ':e actually using iUSMES in their classrooms. These 

districts also tended to have supportive administrators, and more important^ 

ly, a cadre of experienced, enthusiastic USMES teachers who could assist 

their newly trained colleagues. 

Chapter Summary 

A few major themes can be synthesized from the foregoing discussion of 
USMES teacher training and implementation, esp.<.tUlly of the RPW model. 
These poxnts will be listed below alaag with cut recommendations for im- 
proving iuture training efforts. The USMES d^^r pevs need l:o be clear 
and di^t about the persoxmel, financial, spa^e .r,d time commitments which 
^ USMES sc^^^a^Hnlwidu^^ ^ exWct-OTak^to^USMES. - 

A sizable number of 1973 Summer workshop participants claimed that the 
Design Lab, "hands-on," and interdisciplinary aspects of USMES wer^ used 
as "bait" to attract educators to the program, but the demands of local im- 
plementation were not addo^assed carefully. Schools and individual teachers 
who ^.re uncertain about rite nature of the program, ^who had ^misguided notions 
abovf^^tie support they wamid receive, or who did not understand what was 
expec,-^ of them, soon became disinterested or even disgruntled. The USMES 
developers have responded to this suggestion by addressing the issues about 
comnitments in their newer written materials, in their application pro- 
cedures for workshop participants, and in other written and verbal media. 

208 



-170- 



Who teaches the USMES workshop sessions on at any level, was an issue 
of prime concern to' all of the groups of respondents whom we interviewed, 
regardless of their position. Teachers, team leaders and administrators 
alike urged that workshops be taught by very dynamic persons, who pre .knowl- 
edgable about the USME.S approach and the content and skills of various units. 
They should also be able to communicate this knowledge effectively and un- 
derstand the problems faced by elementary school teachers. Not surprisingly 
then, the USMES developers must deal with the perenially difficult problem 
of identifying and using outstanding instructors who fit a variety of de- 
manding specl£ications« 

Teachers alsc asked formore opportunities at workshops to discuss prob- 
lems about - :.W^om application of USMES with experienced USMES teachers. 
Working on lR]SME5 «.--th 30 or more children, especially in the Design Lab, 
was an issue wbH^b concemed-teaeher-s-greatly, and they found informal 
discussion -with peers who have had to deal with such problems very helpful. 
-=OW=tmP^^t ion r0-th^»SM&5=e&nt-ra-t=&ta.f their 
policy of .cWneiling all their most experienced, committed and competent 
USMES teac),er-s into future unit development work. We urged instead that 
they use 5 .mr vr most of these teachers as trnmbers of district resource 
teams. CAyen zhe relatively advanced stage of development work, we pointed 
out .that viofrHpt^ead implementation and maintenance had become more pressing 
issues. Our g^p«ral suggestion brought interested, studied and somewhat 
receptive r^s.p.:-:-^es from cerxain members of the Central Staff. Those who 
were primarrHy responsible for directing unit developing work clearly were 
not favoraTsle to the suggestion. 
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Organization of the USMES workshops rather than the content per se 
seemed to concern many participants. Most teachers agreed that experi- 
encing USMES units the same way their children would was the best way to 
learn about the program. However, they would like to. use several units . 
in lesser detail rather than to use one unit so intensively. They felt 
that this distribution of time over a greater number of units would make 
them more knowledgable about the program and more comfortable with its 
use. The method didn't transfer to other units with experience on only 
one unit. Acknowledging this issue, the USMES Central Staff has been ex- 
perimenting with a variety of organizational schemes for their national 
workshops subsequent to these held during Summer, 1973. 

The USMES Evaluation Team is hopeful that these findings, commentary 
and recommendations on USMES teacher training will be helpful for improv 
ing future training efforts at the national and local levels. 
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» CHAPTER VII 

INDIRECT EFFECTS OF USMES IMPLEMENTATION 

This chapter reviews the observed effects of the USMES implementation, 
both positive and negative, on personnel other than the subjects of the pro- 
gram. The primary outcomes-the effects of USMES on studeiu' s -■ bee- 
considered in Chapter IV. In addition, the viewpoints of teachers and ad- 
ministrators have been presented as indicators of program utility and pro- 
gram effectiveness. However, what we are reporting here is the effects of 
the pr.ogram usage on these same teachers and administrators. These observa- 
tions were made during site visits or were collected via interviews. 

Effects on USMES Teachers 

Teachers using USMES seemed able to stand back, observe, and review their 
own role as teachers. Many found they had grown dissatisfied with the tra- 
ditional role of teacher as the authoritarian person from whom all order and 
"^erl emanate.~YerTh^rf^nkly realized that if an alT^rnaTive rol^h-id 
not been presented to them, they would never have made this discovery. They 
found that they had grown far more respectful of children and children's 
opinions, more sensitive to the way children learn, more aware of how little 
attention they had paid to these factors previously. This discovery has been 
made, not only in their own classrooms, but in the opportunity afforded by 
USMES to interact with other teachers at workshops. Another large segment of 
the teachers-at least one third-were aware of these alternative teacher 
characteristics in themselves, but explained that this was not a result of 
USMES..... they had always behave* in this manner. 

r 
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A few USMES teachers, especially those located in urban areas and 
faced with heavy pressures for standardized test score increases, were 
caught in the midstof conflicting commitments. On one hand, they had 
to answer to the demands of parents and school administrators, and on thr 
other, they were urged by USMES developers to -honor -their^t£ain^ng^ commit- 
ment and ^o implement the program in their classrooms. The priorit>of^ 
iconrnitments was not cLeax to them and they resented being caught in this 

doubl e bind. v^-- * - • - * - - - ^ ^ c . . ^ -i. . ^^^.t .^.j^j; -^l.- 

USMES development:_-t:eachers must write logs of the unit development ac- 
tivities which they create, nurture and oversee in their classrooms. Most 
enjoyed the responsib^ity of classroom unit development but abhored having 
to write the logs, even though they were being paid for their efforts. They 
were particularly discouraged when, after sending their logs to the USMES 
Cen^tral Staff, they did.^ot receive any comments or constructive criticisms 
on their efforts. The ^development teachers ask for continual feedback on 



comment from the development teachers. 

Effects on Non-USMES Teachers a nd Their Students 

In an earlier chapter, we commented on the problem which resulted from 
the location of control classes in the same schools as the experimental 
classes™. The .'contamination" of these non-USMES classes by the USMES pro- 
gram was a phenomenon which members of the evaluation team observed during 

theiir :sifee visitations. 

However, from another perspective, this stands as a positive factor: 
the philosophy and op^aXion of USMES is inviting and infectious to non- 
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USMES teachers and students. One non-USMES teacher borrowed ideas and 
materials from an USMES coUeagueUnd pursued a crs^ lenge in b own cl: . 
Another' read some USMES resource unit books. Several subscribed to a phil- 
osophy which could be paraphrased as the IISMES philosophy. Students who 
had been trained in USMES fen previous yeaJis persuaded their non-USMES 
teachers to allow them to use the Design iab. 

These are but a few examples of the "^ill-over" of USMES into non-USMES 
classes, students and teacher. They were gleaned from our site visits and 
are, only partially indicative of the full extent of the USMES influence on 
the larger environment. Nevertheless, they do report a positive influence 
of USMES beyond those directly trained and engaged in its program. 

Effects on Principals 

Principals who administrate schools in which USMES classrooms were lo- 
cated, looked favorably on the program but realized decided responsibilities. 
Some even expressed a need to have the extend of staffing, space and admin- 
istrative responsibilities more clearly specified at the beginning of their 
commitment to USMES. Non-USMES teachers requested that the use of the USMES 
Design Lab be shared with their classes. USMES teachers complained that the 
lab was not sufficiently available for their classes-that the lab should 
be staffed by personnel in addition to the USMES teacher, and that it be 
made available for longer hours during the day. :.ther teachers pressed for 
a location for the lab which would make it accesi.lble to all rather than to 
the USMES class alone. In general, the principal was the officer who was 
required to deal with ties« frustrations of the USMES and non-USMES teachers 
alike which arose from the presence of the USMES program. 
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E££ects on Parents 

Teachers and principals reported hearing little or no response from 
parents about the USMES program. We heard reports that some parents were 
willing to supply materials for the Design Labs when called upon, and that 
they were aware of and pleased with the level of interests and involvement 
of their children engaged in the USMES program. However, the evaluators 
had no instrument or intent to engage parents directly; these two comments 
are only second hand and without estimate of their representation or com- 
prehensiveness. 

Rffe cts on School or Cotmnunity 

Again, no deliberate effort was made to measure the effect of USMES on 
the school or community spheres. Obvious to the evaluators was the impact 
of a few USMES unit challenges whose solution extended beyond the classroom 
to the rest of the school, or even into the durrounding community: changes 
in the school procedures for lunch, and recreation; changes in local traffic 
patterns due to a new stop sign. Such challenges, by their design, brought 
about interaction between USMES students and the school/community and affect- 
ed the school/commiinity with their unit activities. 
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CHAPTER VI-I-I 

SUMMARY, CONCLUSIONS-, -RECOMMENDATIONS 



"Proo£ of Concept" - - -^^j^ -''-"-^ ' ■ 

The proof of concept," was to have been the primary focus of this 
evaluation project, i.e.,' the examination of the student's problem solving , 
ability and basic skills as they developed under the influence of r the USMES_ . 
program. However, late funding prevented as intensive a focus upon this 
issue as we had originally planned. Two modifications in the original in- 
tent were made: (1) we salvaged as much as possible of the originally 
planned student performance test data, but we also turned to and .relied^ 
upon the perceptions of the teachers trained and actually involved in the 
USMES project; (2) the issues for investigation were broadened and, despite 
the funder's concern for proof of concept, the project took on the shape of 
a formative evaluation for the continuing development of the USMES program, 
more than the shape of a summative evaluation on the USMES students' de- 
meanor. 

USMES i n Theory and Practice ' 

USMES has been described as primarily a philosophy of education and 
secondarily a set of activities and materials. The USMES philosophy calls 
for real and practical problem solving, problems discovered by the stu- 
dents themselves in their Immediate sUool or community environment, problems 
which are relevant to their own concerns and interests. These students are 
supposed to conduct the necesskry observations, collection of data, represen- 
tation and analysis of data, formulation and trial of successive hypotheses, 
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and dl^on on the final action to be taken. The teacher is to act as a 
collaborator, not as the director and students are to work in groups, 
mutually supportive and not primarily as autonomous individuals. While 
the "disciplinary" nature of the USMES units (or "challenges") is the skill 
and concept of problem solving, they describe themselves as interdisciplin- 
ary in that they draw upon and employ the various basic- skills of the, tra- 
ditional curriculum. i 

This is USMES in theory. Our evaluation attempted to examine, by com- 
parison, USMES in practice. Examining applications of USMES units, we 
found that in the beginning of the USMES unit activity much class time was 
spent in constructing, testing and experimenting, and other "hands-on" ac- 
tivitiis-almost half of the classroom time was spent in these data collection 
processes-but that as the year went on, the USMES classes reverted toward 
the more traditional pattern-of activities: writing, reading, taking part 
in class discussions or presentations, looking and listening to other students, 
but especially to the teacher. Nevertheless, USMES student behavior remained 
distinctive from that of the control students. 

Students in USMES groups spent considerably more time in the processes 
of working and constructing in small groups and proportionately less time 
listening and looking at the teacher. Almost no time was spent in pre-struc- 
tured writing or working in workbooks or worksheets. The results suggest that 
USMES teachers did in fact adopt less directive, less dominating roles, es- 
pecially at the beginning of their USMES units, and therefore students 
assumed more responsibility. 
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Effects of the USMES Program on Studen t Performance 

Our study of the effects of the USMES program on student performance 
was drawn from teacher interviews and from test data. Teachers cited es- 
. pecailly _a .growth in affective skills: students were more cooperative, 
self-directed, inquisitive, more logical in their thinking, and more self- 
reliant. Further, • they liked doing USMES. 

Some teachers also noted improvement in some of the basic skills areas . . j^.^ 
such as arithmetic applications and language arts. Such improvements were 
clearly dependent on the particular units which the individual teacher used. 

Results of the analyses of basic skills data were favorable toward USMES; 
all USMES classes and five of .the control classes realized significant in- 
creases, although USMES classes attained significantly higher scores than 
did the controls. Again, larger, more representative samples and more care- 
■ful data collection with valid instruments are goals for the 1974-75 evalua- . 
tion. Limitations noted in the students performance data collection neces- 
sarily restrict one's confidence in these 1973-74 evaluation year results. 

Materials 

A. Design Laborator y 

From the teachers' viewpoint, the motivating-^rnf luence of the Design Lab 
was enormous, a factor which. offset various complaints and "growing-pains" 
in its usage this 1973-74 year. Initially, it appeared to have been over 
identified as the heart of the USMES program, to the extent that many teachers 
and administrators delayed using the USMES units until a lab had been acquired. 
Only later in the year did teachers begin to see that the lab was not essential 
and that some units did not require its use at all. Other problems centered 
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around the staffing and availability or supplies for the labs, but these 
problems remained administrative, not intrinsic to the concept of the lab 
or its value to learning in the USMES program. 

Teachers reported that students fortunate enough to have access to a 

Design Lab loved using it-^an appeal which in turn strongly motivated the 
teachers. Those in the program looked forward to going; those not in the 
program constantly inquired about it. 
B. Other Materials 

There are few teachers who are satisfied with the materials. In fact, 
they are one of the weakest aspects of the USMES program. The technical 
papers are beyond the grasp of many of the teachers; the "How-To" cards'are 
used by virtually none of the children. The manuals are subjected to a 
variety of complaints: they're poorly orgainzed, too wordy, contain un- 
necessary information. Many teachers feel they haven't the time to read 
them. 

Another aspect of this non-use of the manuals should be noted. We were 
frequently asked by the teachers why, if one understands and accepts the 
USMES, philosophy,, one would make use of a manual? These teachers deemed the 
Manuals too pre- structured and directive. Once a challenge is proposed, the 
teacher follows and supports the interests of the children as they solve it. 
Further, there is the danger that USMES teachers will use these manuals^ as 
they do all teacher manuals, and thereby ignore the unique philosophy of the 
program. 

' This leads us to believe that: 

(a) If the workshops are effective in articulating the philos- 
ophy of the program, the manuals could be very short, giving 
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• suggestions of challenges and possible activities. 

(b) a few well developed challenges would probably suffice 
as examples of good units. 

(c) No matter how well written, the manual will probably not 
be sufficient to direct some peopl-3 to use thfi program in 
the way intended. 

• Several teachers and administrators suggested that the program needs 
good pre-service and in-service education to develop the philosophy, not more 
material s. 
Training Model s 

The USMES project has used a variety of training models to prepare teachers 
for the coordinator's or collaborator's role they must play with USMES units. 
Most of these models have relied on the more immediate impact of an experienced 
USMES training staff at National workshops for preparation of teachers to 
utilize USMES in their classrooms. Thus, part of our evaluation was based on 
the appraisals of national workshop participants. However, special attention 
was devoted to the effectiveness of the 1973-74 Resource Personnel Workshop 
efforts, because this training strategy was bssed on the more cost effective 
method of training district resource teams who, in turn, should train and 
support local USMES teachers. , ' 

The Central Staff needs to clarify the financial, space, and time commit- 
ments which an individual, school, or school district must make to USMES usage 
and especaiUy to USMES training. Realistically, they should expect payoffs 
from local training efforts in the person of second-generation USMES teachers 
using USMES only in districts with already experienced, supportive USMES 
teachers. 219 
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Most of the teacher comments about the training sessions themselves 
applied about equally to both national and local workshops. Below, we 
highlight some of the feelings expressed most often: 

(a) Typically, teachers want to know: "How do you do it in 

the classroom?" There has been some complaint that national 
workshops in particular were more oriented to lecture and 
discussion on the program's philosophy than to hands-on 
experience. 

(b) Teachers asked for more opportunities at workshops to dis- 
uuss problems about cl assroom appl ication of USMES with 
experienced USMES teachers. They valued this exchange 
more highly than "what the 'experts' gave them." 

(c) Wor' 3hop trainers need to be screened carefully r,o make sure 
they fit a variety of demanding specifications. Trainers 
must be dynamic personalities who are knowledeable about 
the USMES approach and the content and skills of various 
units. They should be able to coiranunlcate this knowledge • 
effectively. Above all they should be familiar with the 
problems faced by elementary school teachers. 

Indirect Effects 
A. Teacher Style 

•USMES teachers of our sample were able to objectivize and analyze their 
role as teachers. In their reflections, many felt that they had become less 
authoritarian and less direct as a result of their training and experience 
in USMES. The teachers further reported that they find themselves more 
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respectful of children, their opinions, and the way they learn. 

B. Non-USMES Teachers 

Some non-USMES teachers became aware of the program through an intro- 
duction made at a faculty meeting, while others asked colleagues about it; 
their students had heard of the design lab from peers on the playground and 
pressured these non-USMES teachers to use the Lab. This indirect effect 
also resulted in a "contamination" of control groups who, for this 1973-74 
evaluation, were selected in the same schools as the USMES experimental groups. 
In the future, control classes must be selected from different schools. 

C. Administrators 

Administrators, particularly principals, must deal with various pressures, 
including, the location of space for, equipping, and staffing of the design lab. 
If the principal is not supportive of the USMES program when these pressures 
are applied, he will, in effect, discourage teachers from its continued use. 

Some principals are resentful of the developers for not giving their tea- 
chers more support. On the other hand, the principals report that there are 
no staff competitions, tensions, or hard feelings among teachers as a result 
of the presence of the USMES program in their schools. Scheduling sometimes 
became a problem, especially in the upper grades and in schbols where rigid, 
fixed minute periods are used. There was usually no similar problem in the 
lower grade levels. 

( 

D. Parents 

Teachers and principals reported little or no response from parents in 
regard to the student engagement in the USMES program. When there was a ref- 
erence to the parents, it was positive (e.g.. the contribution of lab tools by 
some parents), 
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E. ^^on1>^^SMES students 

-c:- -tj-SiES studeriL-.n feat is, peers of stud^mt^ engaged in the program, 
WOV.I: ■ -a about and be p^f;^,.-tJ.vely. .impressed the program, either, from. 
:p-avi: USMES cUsaas or fxvm friends present' in the USMES program. Par- 
tici * effects of this extended influence evi:.'^fiiiced thenseives to remiestE 
forus^.^ of the Design Lab by many non-USMES sknc^^ of: she same school. 

F. Sac-ool.s - Communities 

Sc ne schools and a f^ communities became conscious of rthe USMES program 
and experienced the. effects of the students' problem solving activities. 
Those that did were impacted by the effects of USMES unit solutions on the 
existent community and school patterns: lunch periods, recreation, traffic 
patterns. 

Other Observations by the Evaluati on Staff 

A. Characteristics of Successful U SMES Schools 

Probably the single most important condition necessary for the success 
of USMES in the schools is the pr^.PnrP of a very enthusiastic, supportive 

t 

person on the staff . This could be the principal or other administrator. 
These key administrative people are important in getting local training 
started, making resources available and implementing the program. 

Beyond this, there should be f supporting teacher or other member of the 
school staff whose strength of personality and enthusiasm will interest 
faculty enough to attend a workshop or attempt a unit. However, this single, 
supportive person will not guarantee successful use of the probram beyond 
a limited period of time. USMES teachers need the support of each other for 
an exchange of ideas and experiences, especially when the program is not going 
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well. iilhM>>-- tear'-^rs should not be isolatiES. from each other. 

Sec£.nc>*eily'>:-t: is quite important to hsve the space, and iioney to lo- 

•-eate,--eqii\tf5 artdi isiaff- -the- -USMES. design lab. 
B. P-rolxleffl Sft-c ifflt5ons 

AJttHm-- tcheoireticaliy, the USMES program should be adaptaSile to all. 
children. > «c.i^:i»jaiity, it appeared to be uioxe successful in middle class 
schools. is seemed to be more a problem of application than a problem 
intrinsic ? jarogram i:tself, but the problem existed nouBtheless. Sdhrools 

located i Lm^sn: cAass and ghetto neighborhoods reported the most difficnlty 
with the , rar^'J«jiT. Teachers in these settings judged their students not 
responsi23.x.-- sufficiently independent to deal with the program. We sug- 
gest that «£.>rk shops be directed toward using the program with students of 
this socioe-.:Onomic level so that teachers may have first hand experience with 
USMES in this situation, develop more relevant approaches and more appropriate 
expectations. 

Another 3±:^T.culty was the continual, excessive demands made on teachers 
in some dis^iiiircs to rais^ scores on basic skill tests (accouatability) , or 
to try new programs specifically designed to teach more content. Teachers in 
these districts felt (and probably rightly so) that their time should be spent 
teaching the content specified by the district, USMES was initially promoted, 
as ,a math/s:ci.eErc-.e::program. Intentionally or not, the developers convinced 
some: people tltat it would teach math and science in an integrated fashion. 
Sinice it j^id not .do this to teachers' satisfaction, math, and to a lesser 
extent science, were taught in their traditional position and, if -no time 
remained, X^JSES was left out. 
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Some administrators and faculty frankly axka a Urhey don'':-! know what 
the program is supposed to do. They find the d -vellopers vague and evasive 
w&en pressed to respond to that question. Unde> ^^i±is condxtion, the limita- 
tions on time or finances will result in the drc^^s^g of tiie USMES program. 

We suggest that the developers attempt to ^dsef iisE the program more clearly 
than they have so tie schools can in turn decifee earnest her or not they want to 
spend the time of this activity. Content orients©- schooLs nnay wish not to, 
but that decision can be made before a heavy imresistment of t±me and money 
is made in workshops and design labs. 

The developers also need to be more clear and direct :about: the commitment 
school administrations are expected to make to USMES. Teachers and princi- 
pals want to know what they're getting into. Developers and ^especially dis- 
trict people need to say who is going to help, how much, in what ways, and 
what responsibilities must teachers and principals perform in their respectiv 
roles. 
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SGHOQL TitjlTORMATION FORM 1973-1974 



Inst^^icst^. Please fill ou^one form for each school in the evaluation sample. 



1. NairBE: afL.. School: 



2. Adxhz^ss of School: _ , _ 



■■' di.ty/ state 
3. Sstiting (Rfliral, Suburban, .^ity): 



4. AppToxtinat-e -Socioeconomic Level (High, Medium, Low). [This inf ormatioxt 
TTiay be^vaxisble fxom the principal] . ^..^ , — . 



5. Beaarrmasxyr years has USMES been taught in this school? 



liias i gn. Lab. 

Dots tte^cho:ol have: a Design Lab? If yes. please answer 6 B, C and D. 

B. If the Ifeasign L5^ >aas a manager, please list his/her name, position, 
(i,.e. ts^cher, t^^ev aid, volunteer, etc.), iand the number of hours 
he/she ife in iLbei ;Li5it>. , 



mat tOJoLs aanr^-^aTErials ±s the Design Lab. lacking? 



7D.. ilDescrilDie ±Tr::^4ieraL tcSa: Design Lab facilities in the building (in- 

ccludi:n^whe!tiie:r tker^ ifs a separate Design. Lab room, in-class Design 
Lab arrrsngemenlr, pontile Design Lab cart, etc.)- 
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CLASS INFORMATION FORM - USiSS EVALUATION 1973-1974 

Instructions ; Please fill out one form for each class in the evaluation 
sample, US ft fS and control. 



City, State . "SMES - Cantrol? 

School N°- ^ Children 



Teacher 



Grade(s) Unit(USMES) 



Observer , Subject (control) 



USMES Clashes Only 

1. How many years has tine teacher been using .USMES2 



2. What units* other thatn: tie i^aressat ons„ Eias t^ie teacSser aased? 



3. Have any xxf- the chilcdrein iaii fcmnex: e3q)osiiare to 35£^r 

A. If yes I appr.02dbn2Ut*^ly iiDw nsny-:2a5 the :x^hi^ ^ 

B. If yes, what:mn±Es wcere thsT^reasEHied to? ^ — 



4. A. Approximately how ^Mriy hours/we^k were scheduled, foir CU:SMES, including 

the Design Lab worfcT — 

B. Approximately m inany we4ks,. excluiding vacations, were spent on USMES 
this year? 

5. If more than onedtClSae/S vrnxi^TaaS'-'^fiams^ on this 7»*ar>, list the unit titles 
and the approxiinascsiadatses. fm~ the rmit s.. , _ ~ 



6. Does this teachor e3q)ress= am inter esE :in using USMS next year? 
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UmSES and Control Classes. 



1. List the names of the non-USMES science, social studies, math, and language 
arts programs and texts used by this class. __ _ • 



2. List the approximate number of hours spent per week, on the following subjects 
and activities. 



a. 



b. 



d* 



8- 



e. Music 

f . Art 



■ ^ h. Special Projects 



3. How many years has this teacher been teaching? 



b. 



a. Math 

b. Science 

c. Social Studies 

d. Language Arts 



f. 



g. Physical Education g- 



h. 



i. Other (specify: ) i» 



4. List any special training that this teacher has had in math and/or science. 
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Control Classes Only 



1. Has the control teacher used any of the USMES materials this year? Unit 
Resource Books, How-to-cards, Technical Papers, Design Lab? _ . 



2. Have the control children used any of the USMES materials or been exposed 
to USMES activities? — ' 



If yes, please explain. 



3. Does the control teacher use the USMES philosophy in her class? 
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USMES EVALUATION 1973-1974 
Program ^fonitori^;5: Porm 

leacher - ^^^^^ 

Present USMES Unit , 



Other Units done xMs yeax and approximate dates 



School 
Address 



1. How was^ tfe 'TOir ^ou are presently working on introduced to the students 



2. "What were some of the typical student reactions to the unit? 



3. What: were your go^is for this unit? 



How 



did -the students define the challenge for their situation? 
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Did the children lose sight of the goals during the unit? 



If yes, then why in your (Opinion did this occur? 



How 



often and in what situations was refocusing required during the unit? 



Were there fluctuations, in student interest during the work on the unit? 



If yes, please explain at what points these occurred. 



If the children hit any impasses during the unit, at what points did 
this occur and how was the impasse overcome? 



Please explain the nature and extent of any student comments or criticisms 
your present USMES unit and/or the USMES approach in general. 



on 



233 



APPENDIX D 



CLASSROOM ACTIVITY ANALYSIS 



^195- 



CLASSROOM ACTIVITY ANALYST S 



i.cher^ 
lool 



Date 



Observer 



# Students 



USMES/Control 



Unit/Subject 





1 


2 


3 


4 


5 


6 


7 


Measures 
















Count?: 
















Constructs 
















As s emb 1 e s 
















Tests/ Experiments 
















Calculates 
















Records data 
















Writes composition/illustrates 
















Writes (pre-s tructured) 
















Reads How-to Cards; Plays Tapes 






— 










Reads - task 
















Free reading, writing, drawing 
















Messes around with materials 
















CHILDREN 


Talks to another - task 
















Talks to another - social 
















Takes part in small group discussion - 
task 
















Takes part in small group discussion - 
social 
















TEACHER 


Gives pre-structured info to teacher 
















Gives original info to teachers - task 
















Seeks information from teacher 
















Talks to teacher « social 


















Takes part in class discussion or pre- 
sentation 
















Listen/look at child 
















Listen/look at small groups 
















Listen/look at class 
















Listen/look at teacher/ lecture/fi Im 
















Col lectins materials/maintenance 
















J °estinc/wai ting/fooling around 
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Description of Categories - 
Classroom Activity Analysis/USMES 



ACTIVITIES 

MEASURE S; An instrument is used to measure distance, weight, volume or 
time. A measurement is read from a continuous scale. 

Exampl es ; 

Timing with a 'Stopwatch. 

Measuring a bored with a yard stick. 

Measuring the length of a sidewalk with string. 

Weighing a person on a scale. 

Measuring ounces of a soft drink in a measuring cup. 
Measuring amounts with measuring spoons. 
Measuring length with ^ trundle wheel. 

Using a tape measure to measure a person's height. ^ 
Using a classroom clock for timing. 
Using an eggtimer. 

Measruing weather conditions with a barometer, thermometer or 
rain gauge, 

C OUNTS : Quantities or frequencies are counted. 
Examples ; 

Counting the number of pieces of metal which can be picked up by 
a magnet. 

Counting the number of people going through the lunch line. 
Counting the number of white Cuisenaire Rods which equal an 
orange rod. 

Counting the number of cars driving through an intersection. 
Counting the number of squares on a piece of graph paper. 
Counting the number of times a pencil can be sharpened. 
Counting the number of persons with a. particular eye color. 
Counting the number of children with freckles. 
Counting the freckles! 
Counting beans, scissors or books. 
Counting 2*s, 5's, or lO's. 

CONSTRUCTS ; Physical components are put together .to create a whole. Something 
is built or made from scratch. 

Examples ; 

A chair is built. 
A soft drink is mixed. 
A mustery box is made. 
A mobile is made. 
An irrigation system is made. 
Ingredients are mixed. 
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CONSTRUCTS: (cont.) 

Sandals are mixed. 

Something is hammered together* 

An apron is sewed together. 

Life-size puppets are made. 

Wood is cut. 

Cement is mixed. 

ASSEMBLES : Pre-cut or manufactured materials are assembled. A plan, set 
instructions or recipe is followed. 

Examples ; 

A plastic model airplane is assembled. 
A geometric form is made from pre-cut paper shapes* 
A light switch is made from electrical components. 
A barometer is made from a science kit. 
Stamps are glued into a stamp album* 
A jigsaw puzzle is put together* 
. Bones are put together. 
A circuit is assembled according to a plan. 

TESTS/EXPERIMENTS : An experiment is performed and data is collected* 
' Examples ; 

Water is tested with litmus paper.' / * ' 

A soft drink is tested for taste appeal. 

A circuit is tested to see if a light will turn on. 

A chair is tested to see if it is the right size. 

Paper towels, are pulled to test strength. 

A blindfold taste test is run. 

R'ocks are scratched to determine hardness. 

A culture mold is grown on wet bread. 

A bottle of pop is shaken to see if it fizzes. 

Items are dropped in water to see if they float. 

CALCULATES : Arithmetic is done (addition, subtraction, multiplication and 
division). Include math done in math" workbooks. 

Examples ; 

Sums are added. 

D:*vision is done on a Want calculator. 
Frequencies are total ed.. 
Yards are converted to feet. 
Multiplication problems are done. 



RECORDS DATA: A record is made of raw data. 



Examples ; 

The number of people, crossing an intersection is recorded. 
A tape recording is made of noise in a lunchroom. 
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RECORDS DATA ; (cont.) 

Pictures are taken of all the different animals for a report on 
the zoo. 

A record is made of the number of times a die turns up three. 

The height of a person is recorded. 

A map is drawn of an intersection. 

A record is kept of weather information. 

Suggestions are written on the blackboard. 

An inventory is made. 

A description of an experiment is recorded. 
Physical characteristics are tallied. 

wri tes" COMPOSITIC^ /ILLUSTRATES ; An original composition or illustration is 

" ' ' created in connection with school work or a 

class assignment. (Includes graphing when 

the graph summarizes and illustrates findings.) 



Examples : 

A story is written. 

A picture Ls^ painted. 

A book report is written. 

A graph is drawn. 

A play is written. 

Future field trips are mapped. 

Self-portraits are drawn. 

Letters are written in connection with the cl^asa project. 
Social studies reports are written. 
Advertisements are written. 
Essay tests are taken. 

Captain Cook's voyage is plotted on a map. 
A histogram is drawn. 

WRITES (PR E- STRUCTURED) ; Writting is done in workbooks or on .worksheets. 

" ^ Pre-structured questions are answered in writing. » 

Examples ; 

Blanks in a reading workbook are filled. 
A worksheet is completed. 
A poem is copied. 

Spelling words aie written from dictation. 

A questionnaire is filled out. 

References are copied. 

Word definitions are copied. 

A true-false test is taken. 

A map is traced. 

R EADS H0W>-TO CARDS AND PLAYS HOW--TO TAPES ; USMES How-To cards and/or tapes 
. ^ are used. 
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READS --TASK: Reading is done in connection with school work or a class 
assignment* 

Examples: 

A reference book is consulted. 
A text book is read. 

A table of weights and measures is read, • 
A magazine is read. 
Instructions are read. 

A newspaper is read for weather forecast and "current events." 

FREE READING, WRITING, DRAWING ; Free time is used for reading, writing 
drawing (NON-TASKS). 

Examples : 

~ A poem is w:zi:tt:en. 
A letter is written. 
A crossword-^puzzle is done. 
A landscape J2S painted. 
A novel is xsaid. 

-?^SES AROUND WITH MATERIALS : Although the child manipula.tes USMES (oxrnon- 

— ' ' ~ USMES) materials, the purpose of his bekavior 

is not apparent. 



E xamp 1 e s : 

Blocks are piled. 
Clay is pounded. 
Buzzers are rung. 
Etc. 



INTERACTIONS 



The child's predominant activity at the tim^ of observation may be verbal 
interaction titTanotfex person or a group of people. The observer needs t 
discriminate between following categories. 

CHILD TO CHILDREN CATEGORIES ; 

TALK T O ANOTHER - TASK ; The chil_d talks with another child about the task. 

TALK TO ANOTHER - SOCIAL ; The child talks with another child socially. 

TAKES PART IN SMALL GROUP DISCUSSION - TASK : The child talks in a group 
about the task. 

TAKES PART IN SMALL GROUP DISCUSSION - SOCIAL : The child talks in a group 
about social, non-task topics. 
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CHILD TO TEACHER CATEGORIES : 

GIVES PRE- STRUCTURED INFORMATION TO TEACHER ; The child responds to the 
teacher according to a pre-structured format. 

GIVES ORIGINAL INFORMATION TO TEACHER : The child talks with or to the 
teacher about the task. 

SEE/KiS^ INFORMATION FEOM TEACHER ; The child seeks information from the 
:ts[£8a'cher, questions the ^macher. 

ZTABKStl; TO TEACHER - SOCISiS The child talks with the teacher socially. 

' TfiKESEPART IN CLASS DISCISSION OR PRESENTATION ; The, child takes part in 
^.a-ciass discussion or'^ves a presentation to thenclass. 



J^im/ LOOK : 

LUSTEN/LOOK - AT CHILD ; The child attends to anothex child. 

LISTEN/ LOOK ■> AT SMALL GROUP ; The child observes, looks on in a group settir 

■ L ISTEN/LOOK"-^ AT CLASS ; The child observes, looks on during a total class 
activity. 

L ISTEN/LOOK - AT TEACHER/ LECTURE /FILM : The child attends to a teacher, a 
lecture or a film. 



OTHER ; 

COLLECTING MATERIALS/MAINTENANCE ; Materials are collected or equipm'ent Is 
maintained. 

Examples: 

A pencil is sharpened. 

Supplies for painting are gathered together. 

RESTI NG/WAITING/FOOLING AROUND/ ATTENDING TO SOMETHING OUTSIDE THE CLASS; 
— ThT" child is not actively involved in learning or free-time activities. 
The child is phased out or distracted. 
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TO THE. OBSERVER: 

This Manual and the accompanying 
materials consist of the following! 

Instructions to guide you in the administration 
of the Playground Problem 

A catalog of playground equipment 

A form on which to record your observations of 
the children's behaviors 

A cassarte tape for recardxng various segments 
of the sessions. 
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GENERAL INSTRUCTIONS 

The problem solving behaviors of elementary school children constitute 
one of the most important areas for evaluation of the USMES program. The 
Playground Problem is to be used as one means of assessing the success of 
the USMES program in reaching its goals. This test is designed to enable 
the observer to collect data on both verbal and non-verbal behaviors involved 

in problem solving. 

The Playground Problem should be administered to designated USMES 
classes and control classes. Five children are to be selected randomly 
from each USMES class and similarly from each control class in the evalua- 
tion sample. The test is to be given to each group of five children rather 

than to individuals. 

Each group of children should be taken to an open area near the school 
and asked to plan a playground. The materials the children are to use in 
solving the problem, the instructions you are to give them, and the role 
you are to play as an observer will be explained in detail shortly. 

We are interested in assessing the degree of cooperation and self- 
or group-motivated interest the children demonstrate during the entire problem 
solving period and the follow-up question period. We are equally interested 
in the degree to which the children employ practical considerations in solving 
the problem. 

Our analysis of the Playground Problem test results will be based on 
three kinds or records: (a) a tape recording of the children's verbal 
presentation during the follow-up question period; (b) your observations of 
the children's behaviors as recorded on the observation form accompanying 
this Manual; and (c) a layout of the proposed playground which the children 
will be asked to draw on a large sheet of paper. 
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In general, your role as an observer will be to organize the test 
session, to instruct the children on what to do, and to observe and record 
their behavior. Specific instructions for administration of the Playground 
'Problem are given in the following sections of this Manual. 

ORRANIZATION 

1. Selection of Children 

A random sample of five children should be picked from each control 
class and each USMES class in your school. In the past, children have not 
always been picked randomly, and this is not acceptable. When children are 
picked on the basis of good academic performance on the one hand, or on the 
basis of "getting rid of the troublemaker" on the other, the entire session 
will have to be disregarded. 

It would be best for you to pibk the children yourself, but the teacher 
can also make the selections if correct procedures are used. The easiest 
appropriate method is to write the names of each child on a piece of paper, 
throw each piece in a hat,, and then select five. 

2. When to Administer the Plavero nnd Problem Test 

This can be a critical factor. Oftentimes, children are more rest- 
less and less attentive at certain times of the day, and especially at 
certain times of the year-for example,, the day before Christman Vacation. 

Try to run your test sessions at approximately the same time of day- 
that includes the control classes as well as the USMES classes. The recom- 
mended time of day is as close to the beginning of the day as possible. 
Avoid extremely cold or rainy days, since the Playground Problem is to be 
administered outside. 
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second, do not run your test sessions on the day before or after vaca- 
tion periods, or on the days when special school events are to take place. 
In the past, some sessions have had to be discounted because of confounding 

factors of this nature. 

In all of these considerations, use your own good judgement. A test 
administered under somewhat less than ideal conditions is probably better than 
no test returns at all for a class. 

3. Where to Administer the Playground Problem Test 

In preparation for the test, you shoul.I V.- ate a suitable open area 
near the school. An empty lot would be ideal. However, if one is not 
available, a playing field or clear black topped area would be appropriate. 
This area should be the same for all groups of children in the same schools 
on your sample list, both USMES groups and control groups. 

4. Materials tn Accomp any Test Administration 

Prior to the testing session, you will need to gather together the 

following items: 

Observation Equipment 
Observation form 
Tape recorder and blank cassette 
Watch 

Tools (in a card board box) 
50 foot tape measure 
Yard stick 
Ball of string 
Large piece of paper 

Tri-wall (to use as hard surface for drawing plan) 
Felt tip pens 
Pencils 
12" rulers 

Catalog of playground equipment 
Scrap paper 
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^ ^INSTRUCUnNR TO THE CHILDREN 

soon after arriving at the open area, you should give the children the 
following instructions and you should record them on tape: 

"Let's suppose' this area was going to be made into a new playground for 
the children ±11 your school." (Indicate clearly the limits of the area). 
"How woiald you plan this playground?" 

"Here is a catalog of playground equipment which could be bought. If 
■ you had $2,000 to spend, which equipment would you choose?" 

, "Please work together to decide which equipment should be bought. Draw 
a plan of the playground on this piece of paper showing where the equipment 
would be placed." 

"You have forty minutes to work together to make your plan. Here are 
some things you may use if you want to." (Hand one child the box con- 
taining the tape measure, pencils, etc.) "Remember, you can spend up 
to $2,000 on equipment." 

DO NOT GIVE THE CHILDREN ANY SUGGESTIONS AS TO WHAT OTHER CONSIDERATIONS 
THEY SHOULD KEEP IN MIND. In the past, some test results have had to be in- 
■ validated because of suggestions and clues which observers had given to the 

children in the instructions. The instructions should be as similar as possible 
for the USMES groups and for the control groups. Any evidence of intentional 
■ or unintentional bias unfortunately results in invalidation of the test session. 
Let the children know that they will have forty minutes to„figure out 
their plan and draw it on paper. Tell them, that at the end of this period, 
you will ask them questions about their plan, and that their answers will be 
recorded on tapte (more about taping later). 

.V > 
OBSERVATIONS 

During the forty minute problem solving period, stay in the area in view 
;'- of the children. You can repeat the instructions, if necessary. However, you 
J . should not participate in the problem solution by answering other questions or 
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suggesting possible strategies. It is up to the children to decide whether 

or not to use the measuring equipment. Do not demand that any particular child 

help out in planning the playground if he or she does not want to. 

After thirty minutes of the problem solving period have expired, tell 
the children that they have ten minutes to complete drawing their plan if they 

have not already done so. 

During the forty minute problem solving period, the observer should make 
notes on the observation form describing the children's activities. Please 
write clearly. Each activity should be noted under the appropriate category 
heading. These notes should be specific and numbered sequentially. For example, 
under- the heading "Measuring" the observer might note: 

"5. Two kids measured the width of the lot with the 50' tape." The 
number "5" indicates that this is the fifth note the observer has made on the 
observation form. The next note might be: 

"6. One child recorded the width of the lot as 45 feet." This observation 
would be placed under the heading "Recording Data." 

You will have received intensive training in the use of this observation 
form at the Observers' Training Workshop. 

PREPAitATTON FOR TAPING 

After the forty minute problem solving period is completed, you should 
-call the children together to prepare for tape recording the ten minute question 
period. 

Children are often shy or giggly when they first speak into a micro- 
phone. Inaudiable responses make our work of analysis very difficult. To 
get around this problem, please ask each child to recite a sentence into the 
microphone, such as: "This is our plan," or "My name is ..." Tell the children 
that they must speak one at a time, and ask them to speak slowly and clearly. 
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Play the tape back to the children. This will give them some chance to 
get used to recording their voices, and it will give you a chance to see how 
well their voices are being picked up. (Note: this part of the recording is 
not important to us and can be erased)* 

When the entire session is over, we would ike to have the following 
recordings returned to us: 

Pare 1: the instructions as you gave them originally to the children 

Part 2: the ten minute question period given after the thirty minute 
problem solving period and after the practice. taping* 

QUESTION PERIOD 

This period during which the children explain their plan and outline 
their reasoning should be tape recorded in its entirety. The children's 
presentation may be up to ten minutes long . You should record the data and 
group at the beginning of each question period taping. If you wish, you may 
take the children back into the school to make the recording. 

It is very important to remember that the questions you ask the children 
and the procedures you use in soliciting their answers MUST be as similar as 
possible for the USMES groups and for the control groups. • Again, any evidence 
of bias may invalidate the results. 

Although you may have to use your imagination and various strategies to 
encourage the children to respond or to explain what they mean in greater 
detail, use the following "scripf*' . a guide to the specific questions you 
should ask. It is very helpful, we are sure you know, if you show interest 
and enthusiasm in what the children have done. Remind the children to speak 
slowly and clearly so that other people can understand what they have said 
later. Do not rush the children but rather gently encourage them to say what 
they want. ^■ 
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FIRST QUESTION SERIES (Directed to the entire group*) 
~ "How did you do?" 

— "Was it fun?" 

SECOND QUESTION SERIES (Directed to the entire group*) 

^^Explain your playground plan." 

"Why did you decide to but (4) pieces of equipment?" 

— "Do you know how much the equipment you hav^: chosen will cost?" 

"Why did you decide to put the swings over here? The slide over here?" 

— "What kinds of information did you need to help you make your decisions?" 
THIRD QUESTION SERIES (Directed first to the entire group, and then to each child 
in turn who has not yet responded) 

"Were there any other important factors you had to consider in making 
your decisions?" 

"Is there anything anyone would like to say before we finish?" 

While it may be necessary to structure the children's report by asking 
questions, you as the observer should not suggest rationale to the children 
by means of your questioning. For example, if there has been no mention of 
safety factors or indications that the issue of safety has been taken into 
consideration, the observer should not bring it up during the tape recording. 

The playground problem does not have one soluiiifDn. However, in the play- 
ground problem, a certain approach to problem solving is valued. An excellent 
response to the playground problem would include: 

1. Measurement or calculation of available space. 

2. Meaningful use of measuring equipment 

3. Careful consideration of types of playground equipment chosen. 

4. Comparisons between size of equipment as listed in catalog and space 
available on playground area. 

5. Consideration of budget limitations.,. 

6. Accuracy in drawing lay-out of proposed playground. 

1. Consideration of human elements such as safety and aesthetic appeal. 

8. Logical and clear presentation of rationale. 

* When the question is directed to the entire group make sure that everyone 

talks v;ho wants to, not only the "spokesman" for the group. Be sure they talk 
one at a time so that it is easy to understand what is being said. 
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However, particularly on the pre-test, the children may not respond in this 
manner. This in itself is interesting and important data and should not be 
interpreted as resulting from the format of the problem. 

After the testing session is over, review the tape on your own. If 
you think any part of the conversation will be difficult for us to understan^, 

please make a note of what was said and attach it to the observation form. 

Please be sure to return to us all tapings, observation sheets, scrap papers 
the students wrote on, and the playground layouts. The pre-test results 
should be sent to us soon after they have been completed. The Playground 
Manual and Catalog should be retained by you after administration of the pre- 
tests. They should be used again for administration of the posttests. Upon 
completion of the post-tests, please return to us the Manual and Catalog along 
with the testing results for the post-test. 

Instructions for administration of this Playground Problem will have been 
reviewed in detail at your Observers' Training Workshop. However, if you have 
any further questions when you are ready to administer the test, please call 
the USMES Evaluation Team, collect, at (617) 353-3312. 

Dr. Mary H. Shann 
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A MANUAL 
of 

CODING DIRECTIONS 
and 
DATA FORMAT 
for 

PROBLEM SOLVING INSTRUMENTS 
(Playground Problem) 
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CODING DIRECTIONS AND FORMAT 

Playground Problem 

On the following pages you will find enlarged representations of the 
coding blocks pertaining to identification, behavioral, cognitive, product 
and validity criteria. The data is to be encoded in the proper column (box) 
In accordance with the instructions accompanying each criterion. 
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Section 1 . —IKiNTl PI CATION (Columns l-ll) 

The objective of the ID code is to identify the teacher, the grade level, the 
Ichool, tL designation and the administration (pre-post) of the test. 

Coding Directions 

1 1 =.r,H 9 fill in the erade code of the teacher. The teachers are 

l:^tZVJot VlW^ slllTL obtained £.0. the ™a,te, list provided by 
the evaluation team office. 

In columns 3, 4, and 5 enter t.he grade as follows: 

Column: 3 4 5 



Grade 1 
Grade 2 

Grade 8 
Grades 1 & 2 

Grades 7 & 8 
Grades 1, 2, & 3 

Grades 6, 7, & 8 



0 0 1 

0 0 2 

0 0 8 

0 1 2 

0 7 8 

1 2 3 

6 7 8 



1 c .nd 7 enter the school code. The schools will be numbered from 
In columns 6 and / encer tne 5,..nuu ^^^^^ nr^)^7ided bv the evaluation 

01 to 99 and shall be obtained from the master list providea oy 

team office. 

In column 8 enter the treatment code as follows: 

I 

USMES 
Control 

In column 9 enter the administration code as follows: 

1 

Pretest 

2 

Posttest 

In columns 10 and 11 enter the unit code as follows: 



Consumer Research - Product testing 
Describing People 

Design for Human Proportion _ ^ 

Electro Magnetic Device Design 
Playarea Design 
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Soft Drink Design 
Traffic Flow 
Weather Prediction 
Burglar Alarm 
Dice Design 
Lunch Lines 
Pedestrian Crossing 
Advertising 
Animal Behavior 

Bicycle Behavior 

Classroom Design 

Communication 

Community Services 

Ecosystems 

Learning Process 

Manufacturing 

Music Production 

Nature Trails 

All control classes should be designated 99 for the design block in columns 
10 and 11. 

Data entered in first 11 columns: ---M/-?.. 



06 
07 

08 
09 
10 
11 
12 
51 
52 
53 
54 
55 
56 
57 
58 
59 
60 
61 











1 1 








1 










1 


1 


2 


3 4 


5 


6 


7 


8 9 10 ii 
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Section l-I. --BEHAVIORAL ASPECTS (Columns 12-15) 

There are four factors which are considered in this segment. The scoring 
of this group shall proceed as follows: 
Factor; 1 

Motivation: to accept the problem and attempt to solve the problem. 
Scoring: 0 No one accepts problem or trys to solve problem. 

1 1 Student accepts/trys to solve problem. 

2 2 Students accepts/trys to solve problem. 

3 3 Students accepts/trys to solve problem. 
^ - 4 4 Students accepts/trys to solve problem. 

5 5 Students accepts/trys to solve problem. 
Enter the proper score in column 12. 

Factor: 2 

Conunittment to task: the level of itensity of the group to continue 

working toward a solution. 

Scoring: 0 No effort. 

1 Disinterested, fooling around, little input. 

2 Some positive input (one or two interested in problem and 
working with little progress). 

3 Group is interested but efforts are not organized, some 
are working on the same item; some factors are not being 
analyzed and time is being wasted. 

4 Group is positively interested and trying to solve problem 
■"" but all actions not useful. 

5 Group is interested, working and not wasting time or effort. 

Enter proper score in column 13. 
Factor: 3 

Organization: allocation of responsibilities for efficiency of manpower. 

Scoring: 0 No effort. 

• 1 Unplanned, haphazard, or chaotic (students do their own 

thing-do not allocate item or all work on the same thing). 
■2 Not all students involved (either by choice or fiat). 

Some are working on problem some are not - may be arguing 
among each other. 
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Students have allocated some tasks - may have some working 

on same item; or possibly 1 may not be involved. 

Tasks are allocated and students working ef f iciently-how- 

ever students'may have trouble with their item and seek 

help. 

Tasks allocated and all are working productively. 



Enter proper score in column 14. 



Factor: 



Structure: 
Scoring: 



Group leadership 



0 None ; 

1 Autocratic--one person dominates who does not listen* to 
other students' ideas. 

2 Minority Leadership--one or two persons listen to others 
and then lead or direct. 

3 Plurality--general agreement of several members leads to 
direction and leadership; most contributions are recognized 
and evaluated. 

4 Democratic--all students contribute; no one's suggestions 
are ignored or ridiculed. One spokesman may arise but 
sources of ideas/efforts are recognized. 

Enter proper score in column 15» 



12 13 L4 15 



256 



-218- 



Section 111. --COGNITIVE ASPECTS (Columns 16-55) 

Data for this section can be derived primarily from the observer form and 
the tapes. It will be necessary to read the observer form and listen to the 
tapes to hridge any apparent gaps or vague statements found in either the form 
or the tape. 

The cognitive aspects shall include variables considered in solving the 
problem and the level or method of measuring the variables. The implementation 
of the measurement in terms of calculation and the recording of the data wxH 

be collected and encoded. 

A total of 10 variables can be accommodated by the scoring protocol. For 
each variable, its identification, measurement, calculation and recording will 
be scored. 
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IIIA. Factor: COST OF EQUIPMENT 



Identification ! 
Scoring: 0 No 

1 Yes 

Enter in column 16« 



Me a s u r emen t : 
Scoring 



0 
1 
2 



No measurement done. 

Vague or very general estimates. 

Estimations by imprecise methods or by eyeballing. It does 
not provide enough information to arrive at a decision, 
useful information which can be used to arrive at a decision 
but the data should be more accurate or precise. 
Precise measurement or clearly appropriate data that can lead 
to solution. 



Enter in column 
Calculations: 



17. 



Scoring: 



0 
1 

2 



No calculations. 

Vague or very general calculations that do little quantification. 
Calculations are imprecise or guesses are arrived at by trial 
and error and are not sufficient to provide necessary data to 
arrive at a solution. 

useful calculations which can be used to arrive at a solution. 
It may not be accurate or have considered totals or balances. 
It should be more precise. 

Calculations are appropriate, precise and can lead to a solution. 



Enter in column 18. 

Recording : 

Scoring 0 
1 
2 

Enter in column 19. 



No records. 

Very general or imprecise records. 
Adequate records. 



1 



16 17 18 19 
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IIIB. Factor: 



SIZE OF EQUIPMENT VS. SIZE OF CHILDREN 
(i.e., larger scale equipment for older 
children; smaller scale equipment for 
younger children) 



Identification ; 
Scoring: 0 Ko 

1 Yes 

Enter in column 20. 



Measuremenj: : 
Scoring: 



0 No measurement. 

1 Vague or general estimates, i.e., big equipment for big kids. 

2 Express need to know proportion of big and small kids in 
their school. 



Enter in column 21. 



Calculations : 
Scoring: 



0 No calculations. 

1 General or arbitrary assignment of equipment for size of 
Sndre-n i.e., for example -lets get half big equipment; 
half small." 

2 More careful estimates on how many big and small kids 
attend their school and selections of equipment reflects 
distribution of size of students. 



Enter in column 22. 



Recording : 
Scoring: 



0 No records. 

1 Very general or imprecise records. 
Enter in column 23. 



20 21 22 23 
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IIIC. Factor: SIZE OF EQUIPMENT VS. AREA AVAILABLE 
(e.g.. a swing will use 100 sq. feet 
and we have 1000 sq. feet all together 
to use.) 



Identification ; 
Scoring: 0 No 

1 Yes 

Enter in column 24. 



Measurement ; 
Scoring: 



0 No measurement done. 

1 Vague or very general estimates. 

2 Estimations by imprecise methods or by eyeballing, does 
does not provide enough information to arrive at a decision. 

3 useful information which can be used to arrive at a decision 
but the data should be more accurate or precise. 

4 Precise measurement or clearly appropriate data that can lead 
to solution. 



Enter in column 25. 



Calculations ; 
Scoring: 



0 No calculations. 

1 vague or very general calculations that do little quantifies- 
tion. 

2 Calculations are imprecise or guesses -^i^J^^^^^/^^^"' 
and error and are not sufficient to provide necessary data 
to arrive at a solution. 

Useful calculations which can be used to arrive at a solution. 
' It lay not Si accurate or have considered totals or balances. 

It should be more precise. 
4 Calculations are appropriate, precise and can lead to a 

solution. 



Enter in column 26. 



Recording 
Scoring: 



0 No records. 

1 . Very general or imprecise records. 

2 Adequate records. 
Enter in column 27. 



24 
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IIID. Factor: CAPACITY OF EQUIPMENT 

(e.g., 4 kids can use a swing 
set with four seats; more kids 
can use a big jungle jim.) 



Identification ; 
Scoring: 0 No 

1 Yes 
Enter in column 28. 

Measurement ; 

Scoring: 0 No measurement. 

1 Vague or general estimates; i.e., big stuff can be used by 
more kids. 

2 Express need to know specific number of children who can 
use each piece of equipment at one time. 

Enter in column 29. 



Calculations ; 
Scoring: 



0 Uo calculation. 

1 General estimates of capacity (e.g., most of the kids in a 
class could use something at the same time)- 

2 Precise figures on capacity (e.g., altogether the equipment 
we choose will handle 25 kids at one time). 



Enter in column 30. 



Recording : 
Scoring: 



0 No records. 

1 Very general or imprecise records, 

2 Adequate records. 
Enter in column 31. 



28 29 30 31 
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lllE. Factor: DURABILITY OF EQUIPMENT 

(i.e., stronger, lasts longer) 



Identif ication t 
Scoring: 0 No 

1 Yes 

Enter in column 32. 

Measurement. ; 

Scoring: 0 No measurement. 

1 Vague statements, i.e., its better. 

2 General/precise, i.e., stronger, last longer. 

Enter in column 33. 
Calculations ; 

Scoring 0 No calculations. 

1 Calculations in a general or vague sense. 

Enter in column 34. 
Recording ; 

Scoring: 0 No records. 

1 Very general or imprecise records. 

Enter in column 35. 



32 33 34 35 
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IIIF. Factor: PLACEMENT Or EQUIPMENT FOR SAFETY CONSIDERATIONS 

Identification t 

Scoring: 0 No 

1 Yes 
Enter in column 36« 

Measurement : 

Scoring: 0 No measurement, 

1 General or vague statements of more or 1:>S3 safety. 

2 More precise measures of safety, i-e., more distance so 
kids do not run into the other stuff. 

Enter in column 37» 
Calculations ; 

Scoring: 0 No calculations. 

• 1 Vague as to placement, i.e., that close enough. 

2 Some concept of calculation, i.e., about 6 ft. or the like. 
Enter in column 38. 
Recording : 

Scoring: 0 No records. 

1 Very general or imprecise records. 

2 Adequate records. 
Enter in column 39. 
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IIIG. Factor: PLACEMENT OF EQUIPMENT FOR EFFICIENT UTILIZATION OF AREA 



Identificatio n: 
Scoring: 0 No 

1 Yes 
Enter in column 40 • 



Measurement : 
Scoring: 



0 No measurement. 

1 Vague or general statements, i.e., it fits. 

2 More precise statements of placement based on size or shape 
of equipment or terrain. 



Enter in co)uiT^n 41. 
Calculations: ; 



Scoring: 



0 No calculations. 

1 General or vague calculation based on placement and practical 
considerations, e.g., putting it there leaves us with more 
space for playing balK 

Enter in column 42. 
Recording ; 

Scoring: 0 No records. 

1 Very general or vague records. 

Enter in column 43. 



40 



41 



42 43 
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IIIH, I, J. Factors: OTHER CONSIDERATIONS (1), (2), (3) 

Other considerations: provision has been made to identify 3 other con 
siderations. They should be recorded in three separate blocks of 4 units. 
FIRST extra (other consideration) should be coded as follows: 

Identification in column 44 
Measurement " 45 

Records " ' 

SECOND extra (other consideration) should be coded as follows; 

Identification in column 48 
Measurement " ^9 

Calculation "50 
Records " 51 

THIRD extra (other consideration) should be coded as follows: 

Identification in column 52 

Measurement 53 

Calculation " "54 

Records " 55 
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Score each extra consideration (there may be one, or up to three extras) 
according to the following codes for identification, measurement, calculation, 
and recording. 



Identification : 
Scoring: 0 
1 



Measurement : 
Scoring: 



Calculations : 
Scoring: 



Recording: 
Scoring: 



0 
1 
2 



Was an additional variable or factor identified for consideration? 

No ^' 
Yes 

Method used or selected to measure variable. 
No measurement done. 
Vague or very general estimates. 

Estimations by imprecise methods or by eyeballing. It does 
not provide enough information to arrive at a decision. 
Useful information which can be used to arrive at a decision 
but the data should be more accurate or precise. 
Precise measurement or clearly appropriate data that can 
lead to a solution. ~ ; 

Implementation of the method of measurement such as addition 
of* costs or cons id'eratl on of distance/ area^^^ - - - 

No calculations. 

Vague or very general calculations that do little quantifica- 
tion. 

2 Calculations are imprecise or guesses are arrived at by trial 
and error and are not sufficient to provide necessary data 
to arrive at a solution. 

3 Useful calculations which can be used to arrive at a solution. 
It may not be accurate or have considered totals or balances. 
It should be more precise. 

4 Calculations are appropriate, precise and can lead to a 
solution. 

Data is listed and understandable. 

0 No records. 

1 Very general or imprecise records. 

2 Adequate records. 



0 
1 
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Section IV. --PRODUCT ASPECTS (Columns 56-59) 

Evaluation of four product aspects shall be based on the students' drawing 
of their playground design. 
The Product - Plan 
Scale ; 

Scoring: 0 No scale. 

1 Approximate scale that indicated relative size of equipment; 
representationA. of distances are reasonable. 

2 Scale is precise or is coded. 
Enter in column 56. 

Labels ; 

Scoring: 0 No labels. 

1 Labels are present and appropriate to equipment. 

Enter in column 57. 



Landmarks : ' 

Scoring: 0 No landmarks. 

1 . Landmarks are present. A" 
2' Landmarks are present, appropriate and/or coded, i.e., enduring . 
and relevant to playground area. 

Enter in column 58. 
Area : 

Scoring: 0 No area limitations. 

1 Area is defined. 
Enter in column 59» 
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Section V. --RELIABILITY/VALIDITY PROBLEMS (Columns 60-66) 

Based on your review of the audio tape and observer's notes, indica 
whether you think any of the following factors may render this testing 
invalid. Code your response 0 = No, 1 = Yes in the appropriate column. 



"'^"-.^ Problem 


Column 


Biased selection of students 


60 


Prompting by observer 


61 


Prior student experierTce with_thi^^ test 


62 


Inclement weather 


63 


Noisy testing environment 


64 


Outside interference/ interruptions 


65 


Other 


66 
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Section v.- -RELIABILITY/VALIDITY PROBLEMS (Columns 60-66) 

Based on your review of the audio tape and observer-i notes, indicate 
whether you think any of the following factors may render this testing session 
invalid. Code your response 0 = No, 1 = Yes in the appropriate column. 

Problem 2°!^ 

Biased selection of students 
Prompting by observer 

Prior student experience with this test 

63 

Inclement weather 

Noisy testing environment 

Outside interference/ interruptions 65 

66 

Other 
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INTERVIEW FORM FOR USMES TEACHERS 
USMES EVALUATION 1973-74 



-231- 



270 



-232- 



Interviewer: 



Datet 



USMES 1973-74 EVALUATION 
TMTF.RVIEW FORM FOR HSMES TEACHERS 



1, Name* 



2, Addressj 



3. Position (grade): 



4. Number of years teaching experience: _ . 

5. Nature of training/ experience in math, science, social science: 



6. Is the interviewee: (check one) 

a. aii USMES development teacher 
' ^ b. a local RPW trained, second-generation USMES teacher 

7. Number of years experience with USMES: — > 



8. A. USt-lES unit(s) which the teacher is currently using: 



(or none?) 



B. USMES unit(s) which the teacher has used in the past: 



(or none?) 
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I. PI... of USMES - <^ry..nVs Curriculum 



^. would you consider USMES a "P^a-^r s^J^^^^/"^^^^^^^^ " 
subjects of math, science, and social science. 



2. What essential skills does USMES 
a. foster? 



b. ignore? 



Ar^fooratPd aooroach to teaching 
think USMES is really an integratea appro 



3, Do you ..w-.— - jj o 

math, science, and social studies? 



(For RPW trained, "second-generation" teachers") 

^ ^ r. TTqMFS unit in the classroom yet this year? 
4. Have you started an USMEb unic 

a. If YES, continue on with Section II. 

b. If NO, go to Section HI. 
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II. General Ef Fprts of USMES 

A. On The Children ; 

5. Have you noted any changes 
toward math, science, or s 
o£ school, which you think 



(+ or -) in the childrens' attitudes 
cial studies, or any other aspect 
are the result of USMES? 



6. Do the children seem any more res.o^^i)'^'^ ^^g^^^s.^"" 
and/or their own actions as a result of using USMES? 



grams? 



B. nn USMES Teacher_s ; 

8 Has the use of USMES changed your teaching style in any way? (Does 
he/sie see his/her role as a teacher any dxf f erently?) 



t- I r,f fiiscioline problems with USMES 

9. Do you encounter the same kinds of dlscipiin f 

as with other programs? 

(Depending on the information gained answers to question 6, 

you may see fit to eliminate this question.) 



C. On Non-USMES Teachers ; 

10. Do non-USMES teachers In your school ask you .bout USMES! 



U. DO they borro„ USMBS „aterl.ls or express .n Interest In edoptlnj 

your approaches? 
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XII. USMES Teacher's Prepar ation and Training 
12. Why did you get involved with USMES? 

a. volunteer? 

b. requirement? 

c. financial incentive? 



13- Did your USMES training prepare you sufficiently to use 
USMES in the classroom? 



14. What was the nature of your USMES training? 

a. national workshop? 

b. RPW workshop? 

c. other? 

15. Did the training meet your expectations? 

16. What suggestions would you offer for the impro^.^ment of the 
workshops? 

17. DO you continue to get the kind of assistance you need? 

XV. ::^^;^^ r^r 's Reaction ^o^t he^Obse ri^r in the Classro o m 

18. What has been the role of the USMES observer in your classroom? 



19. Has 

way? 



the presence of the observer changed your classroom in any 
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V. Use of Materials 

20. Oftenti.es the success of a new progra. ^-/-;,^^^:,f37^°PeT' 
of useful, ^^^ll^l'l'^^^^^^^^ are 
ift: ::d'rf\hrrf ar™.aterials which you thin, should be 
developed to assist students and/or teachers. 

(Give respondent time to talk.) 



You didn't mention: 

a. Design Lab. 

b. Supplies for Design Lab. 

c. Teacher resource manual (s). 

d. Technical papers. 

e. How-to-cards. 

f . Audio versions of How-to-cards. 

21. Which ones (In each category) did you use, For „hac purposes. 



a. 
b. 
c. 
d. 
e. 
f. 



22 



m regard to the Design Lab. is its use granted as part of a 
reward structure? 



23. Who initiates the students' use of the: ^ 
a. Design Lab? 



b. How-to cards? 
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If time permits and if you are in the school building ask to see the 
Design Lab* 



24. Note: a, size of room 

b. location of room relative to classroom 

c, condition of equipment 



three students (whom he/she 

ASK cne ceaciitii. ^i. ;rw« ^ ^n-.rards. 

will probably select) about the use of the How- to- cards. 



25. Ask the teacher if you could speak to^two or 



Investigate: 

a. Bo the children know: 
- where they are? 



- how to use them? 

b. Do the children feel free to use them at any time? 

c. Do they use thes only at the teacher's suggestion? 
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INTERVIEW FORM FOR USMES RESOURCE PERSONNEL 
WORKSHOP TEAM LEADERS 
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Interviewer: 



Date: 



USMES 1973-74 EVALUATION 
TNTRRVIEW FORM FOR USMF-S RPW TEAM LEADERS 



1. Name: 



2. Position: 

3. Address: 



4. Number and dates of local RPW Workshops conducted: 



5. Number of years experience team leader has had with USMES: , 

6. Nature of USMES experience (former and current) - check as appropriate: 

a. USMES implementation teacher 

b. USMES development teacher 

c. USMES consultant 

d. Participant at National USMES Workshop 

e. Participant at Lansing RP Workshop 

f. Other (please specify): ■ ■ 
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TEAM LEADERS 



Have you conducted any workshops to train local teachers in use of USMES7 
2. A. If YES - when did you conduct the workshops. Why was that ti.e chosen? 



B. If NO - when will you conduct the v^orkshops? 



3. Was the workshop successful? 



4. Did it meet your expectations? The teacher's expectations^ 



5. What would you do differently next time? 



6. What factors are 



crucial in the success of the workshops? 



If not yet answered: 

U „™ „„e ceache.s recruIUd £o. „o,Ushops. (Incentives, Co™p.ls«,n 



2. Di. you set a.e,uate support £.o» appropriate persons, (EDC, A.»inis. 

trators? Others?) 



APPENDIX I 



INTERVIEW FORM FOR PRINCIPALS 
USMES EVALUATION 1973-74 
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Interviewer: 



Date: 



USMES 1973-74 EVALUATION 
INTERVIEW FORM FOR PRINCIPALS 



1, Name: 



2m Address: 




Per pupil expenditure: 



A. Population of school districts ■ — ' 

5. His/her school enrollment (number of children): ] 

6. Number of grades in his/her school: circle: K 1 2 3 4 
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1. What reactions have you had about USMES from: 
a. parents 



b. teachers 



2. Khat «££e« has USMES had on the school environment In general! 



3. Any disruptive effects! (Scheduling! Ne« supplies! Hard feelings 
among teachers?) 



4. Have you had any problems with: 
a. scheduling? 



b. physical accomodations! 



outside of classrooms (i.e. 

Ci ""^ ^'^^ 
design 



policies regarding children working ^^t^ide ^ 
LsiRn lab, out of school) because of USMES? 
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ADMINISTRATORS 



5. Nature and source of financial support for USMES: 
a. Does the school have a design lab? 



If so, who paid for it? NSF or the school district? 



e. What, if any. -nanci^ " are provided to teachers 

who participate in USMES? 



i. by the school district? 



ii. by NSF? 



. H vou receive USMES communications and support? (Attempt 
6. From whom do you receive uoi^ ^,,^9^ 
■licit names of degree of contact?) 



to e] 



• ^ TTCiMPq flata and materials? (Attempt to 



1,1 offer to other principals who 

8. What suggestions if^any. wouH you offer 
are considering adoption of USMES? 
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INTERVIEW FORM FOR CONTROL TEACHERS 
USMES EVALUATION 1973-74 
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Interviewcr: ^ . ^ udce. 

USMES 197 3-74 EVALUATION 

I NTERVIEW FORM FOR CONTROL TEACHERS 
(Or Non-USMES Math-Science Teachers in USMES School) 



1 • ' Name : 

2. Address: 



3. Position (grade): , 

4. Number of years teaching experience: 

5. Nature of training/experience in math, science, social 



6. Number of years USMES has been used in his/her school! . 

?• Number of USMES teachers (currently using USMES) in his/her s 
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CONTROL TEACHERS 



U We are concerned with finding people who might be interested in using 
USMES next year or sometime in the future, 
a* Do you know what USMES is? 



b. If no, do you mean you»ve neve, leard of the USMES; project? (If 
no, interview ends.) 



c. If yes, what do you know about USMES? 
i. philosophy - 



ii. activities 



ili. macerials 



iv. unit names - 

2. What do .you like about USMES? What don't you like about USMES? 

3. Have you shared information with USMES teachers on the approaches used 
the program? 

a. Have you tried any of them? 

b. Were they successful? 

4. Elicit information on competitive ^f eel ing, if it seems appropriate. 



